1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Why does Google still try to crawl pages that I removed months ago?

Discussion in 'Google Sitemaps' started by fraserstride, Nov 18, 2005.

  1. #1
    Hi there,

    Sitemaps is coming up with loads of Error 404 - page not found errors on pages that I removed two months ago.

    My site map does not refer to these pages anymore, and the URLs are not listed anywhere in the website.

    Does it take a long time for Google to stop looking for old indexed pages?

    I originally had all my files in and /html/ folder, but then moved them into the root directory and renamed some of them (the content has changed slightly but is almost identical).

    I think I am being penalised for this somehow, because it won't index my new pages, even though they have been in my sitemap for approx two months.

    Are these pages being ignored totally because Google thinks they are duplicate pages?

    Can anyone help?

    Thanks.
    SEMrush
     
    fraserstride, Nov 18, 2005 IP
    SEMrush
  2. dj1471

    dj1471 Well-Known Member

    Messages:
    97
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    116
    #2
    Are there other sites linking to the old URLs? This will naturally keep Google looking for them for a long time.
    Google will index your new pages much quicker (and stop trying to access your old ones) if you put in redirect rules so that people trying to access the old URLs are forwarded to the correct new URLs. If you're running Apache, you'll find plenty of examples of Redirect rules in the Apache forum.
     
    dj1471, Nov 18, 2005 IP
  3. webstop

    webstop Peon

    Messages:
    218
    Likes Received:
    26
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I think you can use robots.txt to stop google from crawling the pages that you have removed.

    You can get more info here, hope it helps:
    http: //www.google.com.sg/intl/en/webmasters/remove.html
     
    webstop, Nov 18, 2005 IP
  4. dj1471

    dj1471 Well-Known Member

    Messages:
    97
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    116
    #4
    That will stop Google from crawling the old URLs, but it won't help Google find and index the new URLs.

    I recently changed all the URLs on a site. I created a sitemap and put in redirects to the new URLs and now, about two days after changing the site, Google has removed all but one of the old URLs and has indexed all of the new ones.

    Redirects are the answer!
     
    dj1471, Nov 18, 2005 IP
  5. hepy

    hepy Peon

    Messages:
    17
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I had the same problem
     
    hepy, Nov 18, 2005 IP
  6. fraserstride

    fraserstride Guest

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Thanks guys.

    It has re-indexed the new pages so I will just use the robots.txt file to tell it not to look for the old ones.

    Ta.

    Fraser
     
    fraserstride, Nov 19, 2005 IP