Why does Google still try to crawl pages that I removed months ago?

Hi there,

Sitemaps is coming up with loads of Error 404 - page not found errors on pages that I removed two months ago.

My site map does not refer to these pages anymore, and the URLs are not listed anywhere in the website.

Does it take a long time for Google to stop looking for old indexed pages?

I originally had all my files in and /html/ folder, but then moved them into the root directory and renamed some of them (the content has changed slightly but is almost identical).

I think I am being penalised for this somehow, because it won't index my new pages, even though they have been in my sitemap for approx two months.

Are these pages being ignored totally because Google thinks they are duplicate pages?

Can anyone help?

Thanks.

 

Are there other sites linking to the old URLs? This will naturally keep Google looking for them for a long time.
Google will index your new pages much quicker (and stop trying to access your old ones) if you put in redirect rules so that people trying to access the old URLs are forwarded to the correct new URLs. If you're running Apache, you'll find plenty of examples of Redirect rules in the Apache forum.

 

I think you can use robots.txt to stop google from crawling the pages that you have removed.

You can get more info here, hope it helps:
http: //www.google.com.sg/intl/en/webmasters/remove.html

 

That will stop Google from crawling the old URLs, but it won't help Google find and index the new URLs.

I recently changed all the URLs on a site. I created a sitemap and put in redirects to the new URLs and now, about two days after changing the site, Google has removed all but one of the old URLs and has indexed all of the new ones.

Redirects are the answer!

 

I had the same problem

 

Thanks guys.

It has re-indexed the new pages so I will just use the robots.txt file to tell it not to look for the old ones.

Ta.

Fraser