View Full Version : can this be done with google adsense bot
gdtechind
Nov 23rd 2005, 1:26 pm
i want to do something which a few ppl might find odd, but i want to stop all the robots to visit my page, but the googlebot for Adsense only.
i think it is called mediapartners googlebot, but am not sure about what should be robots.txt file for that :confused:
any help is appreciated.
thanks
ozami
Nov 23rd 2005, 2:10 pm
mediapartners-google <- thats the official name. Not sure what the command would be - need to read up on robots.txt commands
EWpro
Nov 23rd 2005, 2:34 pm
User-agent: Robot_Name
Disallow: /dir/
replace Robot_Name with the exact bot name or use wildcard '*' for all.
AFAIK there is no simple way to exclude all but one. You may need to create long list, for every robot except mediapartners-google.
Find out what robots are visiting your sites from your server log and for each bot you want to block add like this:
User-agent: Robot_Name
Disallow: /
EWpro
Nov 23rd 2005, 2:42 pm
User-agent: *
Disallow: /
User-agent: mediapartners-google
Disallow:
The above should block all except mediapartners-google. But actually it fails and block all bots that obey robot.txt - including google! So it may not help your case.
The one in my previous post is rather fool-proof.
gdtechind
Nov 23rd 2005, 3:31 pm
User-agent: *
Disallow: /
User-agent: mediapartners-google
Disallow:
The above should block all except mediapartners-google. But actually it fails and block all bots that obey robot.txt - including google! So it may not help your case.
The one in my previous post is rather fool-proof.
hmm, you said it right, it blocks many other robots as well
will use the earlier code
thanks a lot
elf98
Nov 23rd 2005, 3:38 pm
your robots.txt should contain someting like this:
User-agent: Mediapartners-Google
Disallow:
User-Agent: *
Disallow: /
First we match the mediapartners-google bot and disallows nothing.
After we have matched the adsense bot we match any bot and disallows everything!
jlawrence
Nov 23rd 2005, 3:49 pm
elf98 is correct. Allow first then disallow everything else.
gdtechind
Nov 23rd 2005, 11:59 pm
thanks a lot for that information
it works well
:D
maro
Nov 24th 2005, 12:45 am
Do you mind me asking why are you doing this?
vBulletin® v3.6.8, Copyright ©2000-2008, Jelsoft Enterprises Ltd.