Do I Need ROBOT.TXT on my website??? I see that spider visit my new site and I see that he get code 404. I check and it is no robot.txt on my website. If I need, how to build best robot.txt file All The Best Sunchy
I think you must add one! I had the same problem when I saw in my stats that unsuccesfull hits where done to it so I created one! here is mine A big forums was using this robots.txt so I thought it was good! Digitalpoint has this robots
Hi, No web robot looks for a robot.txt file. Robots search for the robots.txt file. Do you need it ? Many sites do not need it. Do not add one if you do not know why you add it. The only advantage would be that the "error 404" codes would disappear from your stats. This "error" does not mean that there is a problem with your site. It just means that you allow all robots to visit and index all pages. So, for web robots, that's a perfect answer. You should use a robots.txt file only if you want to disallow access to some parts of your site to robots. Why would you do that : - to reduce the bandwidth consumption; - to avoid that search engines visit the same page several times through different addresses. Never use a robots.txt file to "protect" sensible informations. Jean-Luc
It really depends. Is there any folders on your website that you don't want spiders to spider? Like the person above me has said: robots.txt file isn't good to tell where to spider, just basicly tells it where not to spider. I wouldn't depend on it though, because spiders don't have to follow robots.txt, it is just that it's web ediquct <sp>
To get rid of the 404s in your stats you can use an empty robots.txt. Search for and read up on the robots.txt specification, then form your own opinion about what to include in it. Check your logs and see if some robot is hitting you unreasonably and try to exclude it (some robots dont giva a rats a-- about robots.txt so be prepared to fail).