Is 'robots.txt' still important to SEO?

Now that the SE algorithms are more powerful than ever, how useful (or not) is a 'robots.txt' file? Do spiders take any notice of it?

T

 

It's still the only way to tell them where not to go.

 

Its only useful if you have pages you don't want them to look at. Although MSN ignores it.

 

Thanks for the replies.

So if MSN ignores the request in my robots.txt file NOT to access a certain folder, does it also index the pages inside it? If so, then surely the robots.txt file is pointless?

T

 

If MSN is the only bot visiting you then yes, it's worthless.

 

Just to be clear, I don't think MSN intends to ignore it. MSN just has a few bugs. Maybe its only a temporary issue.

 

So the conclusion would seem to be:-

IF (a site has pages you don't want SEs to index)
{
Use robots.txt file
}
ELSE
{
don't bother
}

But don't rely on MSN to take any notice.


That's very interesting and also begs the question of the situation with meta tags. However, as that is a different subject I will start a new thread.

 

Yeh robots txt just use to keep bots out of areas you dont want it going

 

But make sure that you have a valid robots.txt file that allows all bots to visit, as a missing or unreadable file might cause (some) bots to ignore your whole site completely. Example file to allow all bots:

User-agent: *
Disallow:

Code (markup):

Cryo.

 

what are some examples of places you wudnt want a bot to go?

 

Pages full of affiliate links. Pages that redirect to affiliate links.