Yahoo Slurp Crawler.... not playing by the rules?

Whoa... I just caught Slurp in somewhere it shouldn't be...

Screenshot: http://files.unitedbimmer.com/ub.c/komodo/yahoo-printable.png

The link it's viewing is the printthread.php?t=ID for a thread in my forum.

Now, United Bimmer's robots.txt file has this in it first of all:

Disallow: /forums/printhread.php

Code (markup):

And second of all, the ONLY link to that page is from the actual thread itself, and that link has a rel="nofollow" tag in it, so even if I didn't have the robots.txt entry, it still should follow it.

Why/how is it crawling there?

 

If it really looks like this:

Disallow: /forums/printhread.php

Code (markup):

you spelled printthread.php wrong in your robots.txt file...

 

Wow......I quit.

Thanks. Haha, I don't know how I missed that.

 

rooooooooooooooofl good catch minstrel

 

LOL It's the kind of bug you could spend hours staring at, but never notice until someone points out the obvious..

Love the avatar komodo, I'll check out your site when I have a spare 5 minutes this afternoon

 

yep, gotta hate it when you have words running together like that. I have done that when registering a domain before.

Was peeved with myself to say the least.

Brad