I was one of those that had used a certain ad network and got smacked down hard into the supp index during the BigDaddy rollout around Feb 2006. That was rectified and my site was added back into the index with the same rankings for my keywords that I had previously. One thing that I did notice though, was that ever since then it takes much longer for my content to be added to the index. In the past when I would post new content, and it was read by GoogleBot according to my web logs, it would take about 1-3 days to get into the index. Now it takes anywhere from 5-8 days. Page ranking for this content does not seem to be affected, it's just that it takes much longer to get in the index. This is obviously a problem, because when I break a new security story, tons of sites link back to it, but their re-reports of my news make it into the index way before my page. This causes them to grab the lion's share of the traffic for this story rather than it going to me who broke the news in the first place. This is not a one-off event either. This happens every single time I post something new. I post some news. My logs show GoogleBot indexes it. Other people start blogging or creating articles about the news on their sites (after googlebot already accessed my page). The next day these people appear in the index, while it takes another 5-7 days for my page to appear. When my page eventually makes it into G, I have the #1 ranking or very close, but by that point it's old news so does not provide a huge benefit. Also the sites making it into the index are not necessarily higher trafficed sites than mine. Many of them have much lower pagerank. This just happened to me and its driving me nuts because I have no idea how to go about fixing the problem. So thats why I think if I am being penalized, it is a strange one. Amount of indexed pages is fine, and when the content does make it into the index it ranks great. It just takes a long time to get into the index in the first place while all these small mfa/affiliate sites are making it into the index around my news before me. Any thoughts, comments, suggestions, money, or anything else that comes to mind would be appreciated?
Is that link the site you are talking about? I f it is, then the first thhing I would do is go to google webmaster tools, use their sitemap.xml tool and make a site map for the search engines. Next add index-follow and what ever intervels you change your content set a visit to that many days
I have been using sitemaps since a week after they were released. The sitemaps get updated every 4 hours and submitted. They are downloaded often by google. Problem is not GoogleBot crawling the data, but getting the data indexed. Google crawls me all day, it just takes forever for the crawled new pages to enter the index while other sites get into the index in a day.
How often do you update your pages? I noticed the frequency of updating my pages is directly linked to the frequency the content gets cached on Google.
Often..its a fairly dynamic site. Regardless of that though, the fact that I was getting all these inbound links to one story with relevant text on the linking page, should have shown google I was the source and have added me to the index quicker.
I think if the other sites are updating their content more frequently than you, Google is crawling their site more often & as a result indexing their content quicker than yours. Are you creating new pages for these new events or is it on your homepage?
New pages...and remember this is not a crawling issue. Google tends to get my new pages crawled within a few hours The problem is that they are not getting into the index before other sites that are reusing my content and linking back to that page.
Ok. So are the sites linking to you creating new pages or are they using already indexed pages? If they use existing pages, then that might be the reason! Just trying to give some clues as to why this may be happening!
Well if they were indexed pages, then I would not be having problems getting them in the index as they are already there. My problem is that new pages are being crawled but taking a long time to get into the index compared to other sites who create a story based off of mine.
Indexed pages, but the new content is not indexed until the page with the new content is crawled. My point is new content on an already indexed page will get indexed quicker than new content on a page that has never been indexed. This is what could be happening with your competitors.
Ahh I see what you are saying. Some of these competitors pages are brand new like mine. Its a strange one...doesnt really make sense to me
Though my site is dynamic, we rewrite URLs. Therefore to visitors it appears as straight HTML thus Google, or any other search engine, wont have problems crawling the site.
Jolly good... Say no more.... I would just get a couple high PR links to get some deep crawls going. Or if you can post on Craigslist or similar site, this will force a good number of powerful spiders to your site. NOTE: I do not approve of spam, but the trick works.
Appreciate the assistance, but not sure if you read the initial problem post where I explained the problem. There is no issue with crawling on my site. Google crawls 10k+ pages a day, yahoo plenty (obviously not as much), msn so so. The problem is that after Google crawls a new page, it takes way to long to enter the index compared to other sites. MSN and Yahoo add to the index almost within a day if not less. Google takes upwards to 8 days.
In trying to help, I have explained to you above more than once what could be happening. You misinterpret what I say & all you could come up with is "doesn't really make sense to me"!!!! May be you can read through thoroughly what was written above!
Kai, My last post was not referring to you. In regards to your suggestions though, as I said In fact, every one of the sites reposting the information are doing it on brand new pages, brand new blog entries (which are new pages) etc. So they are new pages as far as google is concerned. For those few that are getting into the index with just their home page with the brief desription of the story, and not their story page itself, I can understand where you are coming from. For those, though, that are getting into the index with their brand new pages, it does not make sense to me. Why are these people getting in so much quicker than me when I am the source and they are linking back to me. Btw, on saturday this particular page finally made it into the index and is ranking #3 or #1 depending on the search. So this is not a ranking issue, just an indexing issue and it happens for every new page created. Thanks for your help.