How bad is it to NOT have a robots.txt file

Whenever i check my stats i get 404 error regarding the robots.txt file.

How bad is it to not have that file in terms of ranking of site or seo.


regards,
Puneet

 

so then whats the big deal about them? I am noob, what do they do? why are they there for?????

 

I don't have one and googlebot, yahoo, ask, msn, all of them be creeeeepin.

 

It creates a constant 404 error for bots. It's nice to just get rid of that error.

 

It is not a big deal, just be sure you do not place sensitive information in public directories. Otherwise google will index it and it will be available for everyone forever or so.

 

It is for search engine if you do not want search engine to crawl any specific file or folder then create robots.txt. For example if there any .doc file, specific images etc...

 

ive been wondering the same thing

 

The main reason I use robots.txt is restricting access by searchbots. If you're happy with all searchbots accessing everything on your site then don't worry too much about it - you'll be surprised how many sites don't have robots.txt!

 

Greetings in peace, in divine love and light,
Dear friend. There is no reason why we should worried if the robot.txt could not be found within your site. The implementation of robot.txt within our site is only to restrict or to hide certains files that we don't want a search engines to index it. Nevertheless, many webmasters are believes that to have a robot.txt within our site would only to invite hackers. Yet, the good news is the robot.txt also can be used to block a specific ip address to reach our site and the surprised news is there are tons of website do not have the robot.txt. In peace, in divine love and light.

 

Dear friend,
Greetings in peace, in divine love and light. I am sorry if the info I posted was wrong. Furthermore, based on my own experienced being a blogger that my blog was heavily spammed by spammers that really cause so much trouble and with the aid of the wordpress security feature that suggest the used of robots.txt by adding the spammer ip address has managed to elimanites spam that came from the same ip address yet I haven used the same method on the ordinary website other then the wordpress. In peace, in love and light.

 

Dear friend,
Once again greetings in peace, in divine love and light. I have tested on word press but haven't tested on the ordinary website. You may right that the trick won't work on the ordinary website but it will work on wordpress by following the wordpress suggestions. Thanks so much for the feedback. In divine love and light.

 

I like having a robot.txt file on my server, otherwise Google may index sensitive information, such as passwords. It is not the end of the world if you don't, but I always prefer one

mhclysmic

 

Indeed.

Never put stuff in robots.txt that isn't linked to from somewhere already. robots.txt is to stop the search bots from indexing pages that would normally get indexed otherwise, NOT to keep secret stuff a secret (if you list it in robots.txt it's no longer a secret

If you don't mind the search engine bots crawling the pages you link to, then no need to have one.

Although I'd suggest an empty robots.txt file to keep the server logs clean if you don't actually need one

 

I also keep getting a "404 not found message" when I check my crawler access on Google Webmasters. I host my site at weebly.com. Could someone please tell me how, in simple non webmaster terms, how I can get rid of the 404 error and get all bots to crawl my site? Thanks

 

It is not a big deal, just be sure you do not place sensitive information in public directories. Otherwise google will index it and it will be available for everyone forever or so.