Google crawled error pages !

Discussion in 'Google Sitemaps' started by Selly, Sep 26, 2006.

  1. #1
    I have checked crawl statistics form Google sitemap, I have found few of the pages which is showing "Directory Listing Denied" had been crawled by Google and according to the statistics those pages are appearing for different keywords.

    How to prevent these pages to crawl. Is it necessary to prevent? If not then will it hamper my page indexing?

    Help needed.
     
    Selly, Sep 26, 2006 IP
  2. explorer

    explorer Well-Known Member

    Messages:
    463
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    110
    #2
    explorer, Sep 26, 2006 IP
  3. websitetools

    websitetools Well-Known Member

    Messages:
    1,513
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #3
    Are you sure you/webserver are reponding with correct response headers?
     
    websitetools, Sep 28, 2006 IP
  4. disgust

    disgust Guest

    Messages:
    2,417
    Likes Received:
    133
    Best Answers:
    0
    Trophy Points:
    0
    #4
    http://www.google.com/search?hl=en&q=http+header+check&btnG=Google+Search

    try some of those to make sure. if it's an error page without the right headers, as thomasschulz said, google doesn't have any realy way to know what is and what isn't an error page.

    if they aren't setting out the right headers and you don't know how / don't care to modify them, use robots.txt.

    and keep in mind that google doesn't know how a url will respond (ie, whether it's an error page or not) until it actually tries to crawl it... it shouldn't add it to the index, though, if the headers are correct.
     
    disgust, Sep 28, 2006 IP
  5. mad4

    mad4 Peon

    Messages:
    6,986
    Likes Received:
    493
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Find all the error pages and 301 them all to the homepage.
     
    mad4, Sep 28, 2006 IP
  6. bluesync

    bluesync Peon

    Messages:
    214
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #6
    bluesync, Sep 28, 2006 IP
  7. mad4

    mad4 Peon

    Messages:
    6,986
    Likes Received:
    493
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Yes, but a 404 won't pass the PR........
     
    mad4, Sep 28, 2006 IP