I've encountered a very strange problem on my website. I first started noticing that visitor rates dropped about 15% two days ago for no apparent reasons. In fact, search engine referral traffic was up compared to the same day last week, but visitors are down as evident by Urchin stats as well as actual advertising revenue. Upon some more detailed analysis or Urchin stats, I found that the biggest drop was related to these areas: 1. From browser based traffic, everything seems normal except "Mozilla Compatible Agent (5.0)" - it is down to only 30% of the normal daily average (this drop accounts for the overall visitor drop from the site) 2. External referrals seem normal as well, with most of the drop coming from visits with "No Referrals". 3. Visits from the IP range 74.6.0.0/16 is down quite considerably, from more than a thousand sessions (is this normal?) to just over one hundred. WHOIS seems to indicate this belongs to Yahoo/Inktomi, and looking at the server logs, certainly some of the IP addresses belong to Slurp. Now there hasn't been anything new on the website, no changes to .htaccess or the pages that would explain this. I've even double checked the firewall to ensure it isn't blocking traffic, and it isn't. A Mozilla compatible agent 5.0 with no referrals seems to indicate a web crawler, and the drop in visits from one set of IP ranges belonging to Slurp seems to indicate this may be Yahoo/Slurp related. However, if it is just robot traffic that is down, it doesn't really explain the drop in revenue. Could there be some other service from Yahoo/Inktomi that is feeding me human visitors with the above user agent and IP ranges? On a perhaps somewhat related note, when searching for the name of my website on Yahoo (and it is the first result that comes up), clicking on the "cache" link for my index page gives me a "We're sorry, but we could not process your request for the cache of http://www.mysite.com" error message. When doing a "site:mysite.com", the cache link there for the index page works fine. Any help to solve the mystery would be greatly appreciated.
Doesn't make much sense if you have not made any major changes recently. I do know that there were many reports of a drop in internet activity over the last weekend. My site dropped as did sites belonging to several of my friends. May be some simple self adjusting by web crawlers.
Yahoo it's updating its index as we speak. The crawler is undergoing a major upgrade, links are being re-analyzed, etc, etc. I can see several websites that actually have 1000s of incoming links yet Yahoo only showing a few. On their official blog, Yahoo did advise that it will take couple days for all to settle. Until this process is finalized, there will be some shifting in rankings.
Oh well, never mind. The server is one of those affected by The Planet's Houston 1 datacenter, which is now disabled thanks to an exploding transformer that took out three walls. A bad week just somehow got worse ...