I have run into problems with an old site that has been around since 2000. Currently it holds around 15000 pages but only a third or so are indexed compared to 2/3 or more in the past. The problem is that it does not seem to matter whether the pages hold little or a lot of content (= plain text). Some pages with very little content are indexed while others with long biographic texts (unique) are not indexed. The site has ranked very well in the past for many relevant search words and the indexed pages continue to rank well. We have many links from respected sources such as Wikipedia and business related sites. I can think of the following problems that may cause this reaction from Google: 1. There are no header tags for the main content par of the pages (they are all dynamic with a "shell" around so to speak which contain many links to other parts of the site) 2. There is no sitemap However, since this has never been an issue in the past I don't really understand what might have changed during the past year or so? Any thoughts or ideas on this will be highly appreciated. Also, looking at crawl errors in Webmaster Tools I noticed 1000+ redirect errors but when I test some of the urls listed there they all seem work fine... What does this mean?
Have you made any changes to the URL's of the pages which are no longer indexed or are they all the same? What I've seen before are people make their URL's more SEO friendly but failing to implement proper 301 redirects, resulting in 302 http status errors in the header preventing the Google crawler from accessing these pages. If you don't already know a 302 status means that the page has been "temporarily moved"...unless you implement a proper permanent 301 redirect Google can't be sure that this is going to be the new home of the page or whether it is likely to change again, so it wouldn't make much sense for Google to index it. What are the exact errors displaying in Google's webmaster tools?
If you have no sitemap then u can generate sitemap for ur website because it is important in on-page optimization
Yes, we did make some url restructuring a while back but no problems appeared at that point and the new urls seemed to be indexed just fine. When I check under "Crawl errors" in Webmaster Tools about 1000 urls are listed under "Not followed", the error simply listed as "Redirect error". All of them work fine though... This may be part of the problem but it doesn't explain the rest of the missing pages? It should be noted that the urls listed are not old urls that are 301 redirected but some of the new ones.