Help with robots.txt file

Hello everyone, I have a site www.bignutsracing.com

Could someone be so generous as to instruct me on the proper use of a robots.txt file.

Spiders are allowed to crawl everything on the site, therefore, I have placed the file in both the www.bignutsracing.com directory and the www.bignutsracing.com/forums directory.

All I put in the file was:

User-agent: *
Disallow:

Is this correct?

Any help is greatly appreciated!

 

If you want spiders to crawl everything, you don't need a robots.txt file. robots.txt files are used to restrict certain files/directories from robots/crawlers.

 

Thanks for the response...I thought I read somewhere that spiders automatically look for a robots.txt file and some will not do anything if they do not find one?

 

I would atleast put in a blank page so that the robots find it and it doesn't show up in error logs

 

I could 'google it', but since we already have a thread going please, tell me exactly how to use the robots.txt file to restrict certain files/directories from robots/crawlers?

Thanks

 

if i want to restrict certain bots then what should i write in rorots.txt file

 

Hi,

The first step is to read the original specification and documentation (this does not require a PhD in computer science).

Jean-Luc

 

I added a robots.tct file to my site but am not sure what it does or if it is doing anything but according to google site map beta it looks at it every few days. So it must do something

 

User-agent: BotName
Disallow: /folder1/
Disallow: /folder2/

say, for example, you dont want google to crawl your images directory. The usage would be:

User-agent: Googlebot
Disallow: /images/

If you want to block all search engines:

User-agent: *
Disallow: /images/

You can get detailed tutorials if you search on google

 

Here is a great article that explains the robots.txt file.

http://www.marketposition.com/blog/archives/2005/09/controlling_sea.html

 

We had restrict robots.txt for a while time. Is any chance to say googlebot to index our website now? We change robots.txt to allow indexing website, add website via form at Google pages, but googlebot do not indexing yet

 

Google Webmaster Tools can help you solve this. Create an account if you don't have one, then add the site to the account and ask Google to verify it.

When this is done, go to "Tools", then "Analyze robots.txt" and Google will read your robots.txt. Then you will have to be patient as the following actions by Google depend on the number of quality links pointing to your web site.

Jean-Luc

 

OK, Thanks. I try it. We have Google webmaster tools account.

 

That is correct but you don't have to put another robots.txt in the forums directory

w w w.bignutsracing.com/robots.txt will suffice for all the directories under it