Final Fantasy - Mobile Phones - Debt Consolidation - Mobile Phones - Credit Cards
PDA

View Full Version : can this be done with google adsense bot

gdtechind
Nov 23rd 2005, 1:26 pm
i want to do something which a few ppl might find odd, but i want to stop all the robots to visit my page, but the googlebot for Adsense only.
i think it is called mediapartners googlebot, but am not sure about what should be robots.txt file for that :confused:

any help is appreciated.

thanks
ozami
Nov 23rd 2005, 2:10 pm
mediapartners-google <- thats the official name. Not sure what the command would be - need to read up on robots.txt commands
EWpro
Nov 23rd 2005, 2:34 pm
User-agent: Robot_Name
Disallow: /dir/

replace Robot_Name with the exact bot name or use wildcard '*' for all.

AFAIK there is no simple way to exclude all but one. You may need to create long list, for every robot except mediapartners-google.

Find out what robots are visiting your sites from your server log and for each bot you want to block add like this:

User-agent: Robot_Name
Disallow: /
EWpro
Nov 23rd 2005, 2:42 pm
User-agent: *
Disallow: /
User-agent: mediapartners-google
Disallow:

The above should block all except mediapartners-google. But actually it fails and block all bots that obey robot.txt - including google! So it may not help your case.

The one in my previous post is rather fool-proof.
gdtechind
Nov 23rd 2005, 3:31 pm
User-agent: *
Disallow: /
User-agent: mediapartners-google
Disallow:

The above should block all except mediapartners-google. But actually it fails and block all bots that obey robot.txt - including google! So it may not help your case.

The one in my previous post is rather fool-proof.

hmm, you said it right, it blocks many other robots as well

will use the earlier code

thanks a lot
elf98
Nov 23rd 2005, 3:38 pm
your robots.txt should contain someting like this:

User-agent: Mediapartners-Google
Disallow:

User-Agent: *
Disallow: /


First we match the mediapartners-google bot and disallows nothing.
After we have matched the adsense bot we match any bot and disallows everything!
jlawrence
Nov 23rd 2005, 3:49 pm
elf98 is correct. Allow first then disallow everything else.
gdtechind
Nov 23rd 2005, 11:59 pm
thanks a lot for that information

it works well
:D
maro
Nov 24th 2005, 12:45 am
Do you mind me asking why are you doing this?