Hi, This is very urgent as I am now starting to panic like crazy. Well this is how it goes. I have a large site that includes about 4 sub domains. Well I had found that one of the sub domains had got hit by someone scraping my sites content last week or they were doing something as the bandwidth and hits from two seperate IP's were large that I decided to find out more about the IP's and thought that it would be best to ban them so that they could not access my site using them IP's again. The IP's are: 64.53.209.126 24.192.83.240 Between both of them they consumed about 4 gb's of bandwidth between them in just about 2 days. I thought that was an extreme amount and had no choice to ban them, nowing that they were not from Google anyway. Since I had band them, my rankings within Google for this domain have plumeted so much that I am no longer getting any traffic from Google for this domain. Every other sub domain is fine it is just this one and infact this sub domain was getting the most traffic. It was ranking at 3rd position on google and better for many keywords and now it is ranking very poorly. I have even noticed that alot of the pages from my site has gone into Google's Supplemental results pages. I am very worried as to what to do. Do Google use these IP's. I checked and it seems that they are from the US, I think, but it didn't tell me that they were the google bot. Please help as I want this sub domain to get back to the top as this is loosing me alot of revenue as infact it was were most of my revenue was being earn't. Thanks and this is urgent, so that I know whether to unblock the IP's or not. If they are used by Google then I will unblock them, but if they are not then I don't want someone else getting all my content so easily and also slowing my site done and costing me money in bandwidth. Thanks!
I have used it, but it just gives me the location of the IP, how do I know if it is Google's or not? There is nothing saying who is using it, what site's using it etc. I need to know if it is used by Google bots or not. Please help!
I had read about this in Matt Cutts blog - it goes something like this - if i remember it well This methos only relies on Google having properly set up DNS entries for the crawling IP's. It is a two step process and involves doing a reverse dns lookup, then a forward DNS lookup. For eg: It works out the same way for the other IP as well.. u can try using DNS-stuff shows no signs of googlebot in it atleast
Does Google have some sort of tool so that you can check if they can crawl your site properly. If so, then I need to use it quickly. I am loosing hundreds of dollars a day from this. Also, if anyone knows how or where I can find out if them two IP's are Google's then please let me know so I can un ban then straight away. It just seems strange that I ban these two ip's last week and this week my rankings have gone plus alot of pages have gone into supplement index. These rankings have been where they are for around a year now. I am really worried about this.
http://www.seonewthing.com/googleBotCheck.aspx this uses same method as I did.. but I think thats the only reliable way to check it
Neither of those IPs belong to Google. They are from Mt. Prospect, IL and they are blacklisted as spammers. Show us how you blocked them(probably .htaccess?) maybe you made a mistake. If you PM me your website I can find out why you dropped position in the SERPs, as well as any other red flags. I won't be on again till the morning though.
Use http://www.us.sorbs.net/lookup.shtml They blacklist for spam/phishing/open proxy abuse and a few other things, so you are looking up an IP for a wide range of abuse. It's backed by Cisco, Sun, Google and others.
Thinks are getting a little more worrying now. I think I would have prefered it if these domains were Google's. At least I could have unbanned them and my listings would have gone back. But now it looks as if they are not and my site is getting properly crawled by google as I have just checked. They have some tools so that you can check your sites ranking for certain keywords, if google has crawled your site properly, what pages it had problems crawling, when it last visited your site, etc. I only just found that out, but would like to know that when you add a site, do you have to add the subdomains aswell as if they were a new site or would the main domain account for them. I am on about adding a site to the Webmaster Tools part in my google account.
Ummm, Thanks for the link. I have checked it out and it was the right decision to block them two ip's.
Here is my .htaccess part of the file where I have blocked the IP's Is this correct as it was done through my servers control panel, so I didn't directly write it in the file. I have looked on the web and it shows then doing it like: So has my server write the .htaccess file wrong.
One thing we always do is to think that whatever we do has an immediate effect on our Google results. Google has a lag that is 1-2 months so whatever we do today will not show up in Google until 1-2 months from now, so you blocking IP addresses doesn't have anything to do with your supplemental pages. Yesterday Google did an update and my pages that has been in supplemental hell for months, suddenly came back with tons of traffic as a result. From posts here we can see that when someone gains traffic, someone else looses traffic. You need to check your subdomain and try to figure out why they might have "punished" you. As for checkin Googles crawl you can use their sitemaps. I don't use the sitemaps at all but I have the account so I can see if there are any problems.
www.arin.net or "dig -x <ip>" in linux. Arin will tell you who owns the block that is crawling. Dig -x shows you reverse DNS, so you'll see something like : 228.66.249.66.in-addr.arpa. 86400 IN PTR crawl-66-249-66-228.googlebot.com. Get similar things for all the major SEs. In Arin, you'll see Google owns the IPs and it'll tell you the block. If you understand CIDR notation, you can block a spammers whole block, a single IP isn't gonna help a lot. In .htaccess : Deny From 1.2.3.0/24 would block 1.2.3.0 -> 1.2.3.255 Arin will tell you what block the spammer owns. On some of my domains I've have blocked /8's from Asia because they spam so much. But /8 is a LOT of IPs, so be sure not to use CIDR notation if you don't know what it means. Hope that helps!
It just seemed rather strange that early last week I banned the two IP's and then by the middle of this week my rankings disappear. This is a site that was not put up yesturday. It has been online for over 4 years and the sub domain that has lost is rankings is infact the most popular part of the site and also had the best rankings and also the most and best backlinks. All the other sub domains on the site ranks just the same as they always do. I have done no spamming, I have not do anything I shouldn't have. Alot of the links pointing to this site is also automated in that people like it and automatically link to it from their site. I also don't do link exchanges. I hope it will recover very soon. I am also looking at the IP deny script, it seems that my server had written it wrong and it should be done differently. Although I still think that because it has been written this way, it still shouldn't have stopped googles robots.
Like I said, we too often connect something we do with immediate results in Google SERP's. This is not the way it works. With MSN it takes about a week to see results, but with Google you never know. Something you did today might effect your listing in Google in 1-2 months. You banning IP's a couple of days ago has nothing to do with changed ranking. Google are just now changing the results and I have seen it the other way round. My site disappeared in june, came back in august for a month just to disappear again. yesterday it suddenly came back again and I am not sure why they decided to get my site back again. Maybe it was something I did 2 months ago and I am just hoping that nothing I have done during the "waiting period" to screw up my site so they dislike it again in the next update in a month.
I guess they must just dislike my site. I can't see that I have done nothing wrong. Even if my site was submitted to alot of bad sites, it still should have not done any damage as if this was the case then my competitors will be submitting my site to bad sites on a daily basis. I guess it may be the slow speed of new links pointing to my site compared to the growth in content. I have not done much advertising over the past few months, but then I have had alot of people automatically linking to my site, but while all this is been going on I have had quite a bit of new content added. I guess when I get time I will have to take it a little further and start getting some real media and back links with press releases and other forms of advertising.
If you could PM me the URL I will try and analyze the causes for your fall .. you can add me to your IM as well..