robots.txt

[joeychgo]

ok - I dont have a robots.txt - basically because I dont have any areas I dont want the spiders to crawl that my forum software would let them. (im not even sure what to put in one)


How important is it to have a robots.txt file? Also -- could it be used for KW optimization somehow? Maybe placing KWs in the robots.txt file ?

[Cricket]

I wrote an article some time back that may help answer your questions.

Create Robots.txt File (http://www.gnc-web-creations.com/creating_robotstxt_file.htm)


Cricket

[cyanide]

Technically you don't need a robots.txt, because robots are programmed to follow links and crawl websites.
however you could say, it's good etiquette to have one. The good bots will automatically look for one, not having one may produce a 404 error

Good tutorial, Cricket

[quadcity]

Hello joeychgo.
>How important is it to have a robots.txt file?
If you don't need to exclude anything, the only advantage is preventing all of the harmless robots.txt 404's in your log file.

>could it be used for KW optimization somehow?
No.

[Smyrl]

Beautiful site, Cricket.

[john_loch]

Yeah Cricket,

Pretty site - I like it :)

[GiddyGamer]

Thanks to Cricket for her referenced article. I found her robots.txt validator link is now defunct/redirected to a paid service. Googled for a new one and found this one quite simple and speedy:
http://www.invision-graphics.com/robotstxt_validator.html

Per advice in this thread I have added a blank robots.txt, and validated it - thanks!

the Giddy Gamer
www.giddygamer.com