We get hit pretty hard from a bot that calls itself AdsBot-Google+(+http://www.google.com/adsbot.html) with the IP: 72.14.199.65 . Here is the reverse DNS on that IP: http://www.zoneedit.com/lookup.html?ipaddress=72.14.199.65&server=&reverse=Look+it+up It all says Google up to this point. But then I try doing a forward look up and the tool doesn't find anything at all. We have a * crawl delay in the robots.txt file, but this bot doesn't seem to be paying any attention at all. User-agent: * Crawl-delay: 120 Can anyone verify that this is or is not a google crawler? And if so, how did you go about it? Thanks!
AdBot is AdWords, not AdSense. It checks landing pages for quality control. Do you run AdWords ads at all pointing to those pages? It looks like a Google IP - used by FeedFetcher before.
Sorry, yes I clicked on Adsense out of habbit. We do run Adwords. That's why I'm worried about blocking it because I don't want our relevancy score to go down the toilet. When I'm in webmaster tools checking the robots.txt file google says they just ignore the crawl delay.
A few months ago a Google spider hit some of my sites and used 40-50 times the entire size of the site in bandwidth. It went on for two or three nights, stpped for a day or two, happened again, and then stopped. It's hard to know what to do since I hated to block Google
mhdoc - are you sure it wasn't someone spoofing that user agent? You're supposed to be able to do a reverse DNS to check for the host / domain, then a forward DNS to check the host with the original IP. It's kind of like double checking your math answers by doing the problem in reverse. The only thing is, I can't get any info from the forward lookup. Perhaps I'm doing something wrong. Anyone?
It *is* the Adwords bot checking out your relevancy. Thing is, it does it for *every* keyword in *every* adgroup, and every time you make a change it comes back to make sure it hasn't missed anything!!! Aaaaaaarrrrrggggghhhhh! (no more 100,000 keyword campaigns for me, thank you!) H
Thank you Barry. I see you at the conferences and read your blog, and I know you are on top of things. Thus, I do not take offense to the fact that you assume we have not read the instructions on that link over a dozen times. You will notice that in my first post I talk about doing a reverse DNS followed by a forward DNS. Perhaps I was doing it wrong, but the methods described at Goog Webmaster blog and the one recently by MSN do not seem to be working for me. That's why I came on here to ask for help. I do appologize if I sound like a guy who doesnt appreciate when someone is trying to help him, as that is not the case. Any and all help is appreciated. So for now I'm going to assume that it is indeed Google checking out my landing page relevancy. In that case, is there a way to get them to follow the crawl delay command in my robots.txt file? Kind Regards, Everett
Your pages are PHP I see - I started a thread here on DP in the PHP forum yesterday to assemble code that does the DNS lookups. Maybe that helps you verify them. It may well be rogue bots. Otherwise, you might have to move to a bigger bandwidth server - maybe you are outgrowing your site's home.
Are you looking at the fitness marketing or fitness directory sites? I'm sorry, I should have explained. Those are more my hobby sites than anything. This is for a client that runs a health and fitness ecommerce store. We get about 300,000 unique visitors per month. I'll go check out that post you mentioned. Thanks TOPS3O!