Disallow

Hi Guys,

How can I disallow this link from Yahoo?
blogs/join.aspx?ReturnUrl=/members/

I have this on my robots.txt file

Disallow: blogs/join.aspx

but yahoo is still crawling this blogs/join.aspx?ReturnUrl=/members/

can someone help me out?

Thanks,


Frio

 

google it you will fid the string which you need to specify

the method is correct

 

How long was the gap between updating your robots.txt file and Yahoo crawling the page ?

In my experience, Yahoo has one of the highest page -> robots.txt request ratios I have seen. The only one I have seen higher was Baidu. This means that it might be a day or two before it notices that your robots.txt has changed.

Google have multiple bots and they don't appear to share information. This means that even though one of the bots has seen your new robots.txt, the others may not have and will keep crawling. Yahoo may also do the same thing but I haven't actually observed this behaviour.

 

Try writing the below code...

User-agent: yahoo
Disallow: /blogs/join.aspx

Code (markup):

I think you have missed the forward slash(/) sign which indicates the root.
Cheers mate... Peace!

 

+1 SwapsRulez

 

What I heard from an SEO guru is that Yahoo may not obey robots.txt in your site. But, Google, yes, it does.

 

its not true.... but in some cases.... the advanced syntax like using * or $.... will be the real pain for the yahoo slurp to understand the meaning of them.... instead the google & MSN bots understands them easily...

 

This should work

User-agent: yahoo
Disallow: /blogs/join.aspx
Disallow: /blogs/join.aspx?ReturnUrl=/members/​

But, the problem you are having will be the same for all SEs... I would block all of them with:

User-agent: *
Disallow: /blogs/join.aspx​