Google blocks part of my site because it's "not found" ???

Discussion in 'Google Analytics' started by cristian, Oct 1, 2009.

0
  1. #1
    Hi guys,

    I have a site at 69.162.104.186 and Google's bots have been routinely stopping by each week for months now. And for months now, AdSense Site Diagnostics advise me that 69.162.104.186/gallery is a blocked url because it's "not found."

    Any ideas on why this may be happening? I'd not be overly concerned about this, if it weren't for the fact that literally everything on my site is in this directory. I have hundreds of pages of original content that I'd like folks to be able to find.

    Cris
     
    cristian, Oct 1, 2009 IP
  2. Deus Ex

    Deus Ex Peon

    Messages:
    531
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Check your robots.txt file and make sure all pages are indexable by the search engines. Also, make sure that there is a sitemap and other aspects of on-page/on-site optimization for SEO
     
    Deus Ex, Oct 2, 2009 IP
  3. theapparatus

    theapparatus Peon

    Messages:
    2,925
    Likes Received:
    119
    Best Answers:
    0
    Trophy Points:
    0
    #3
    You didn't even look at the url, did you?

    When you view that url, you'll note that what you get is just a list of files instead of actual webpages.

    The google bot may be seeing it as a file directory instead of a web directory since all that's there is files and not an actual site. What you can do though is view the directory, view the source of what you're looking at in your browser and save that html as a index.html file within the /gallery subdirectory. That way the bot will know that there are actual pages that need to be indexed instead of just files that need to be stored.

    Hope this helps,
    -drmike
     
    theapparatus, Oct 2, 2009 IP
  4. cristian

    cristian Peon

    Messages:
    16
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    LOL! Actually I spent way too much staring right at that url and trying to figure out why a bot would see a directory full of html pages and think it was really just a bunch of stuff I had on my site and didn't want them to bother with.

    I guess the thing I should've done from the start was to just move the gallery_index file right into that folder so as to not confuse the spidering process.

    Thanks for the info on this!

    Cris
     
    cristian, Oct 2, 2009 IP
  5. theapparatus

    theapparatus Peon

    Messages:
    2,925
    Likes Received:
    119
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Please use the rep button (That's the one with the scales at the upper right of each post) as the mods here don't take too kindly to "Thanks!" posts.

    What's interesting about this question is I spent about an hour yesterday looking for a reference to if the Google bot would index an subdirectory like that and couldn't find a thing. I remember discussing it previously a long time ago but nothing came up.

    You would have thought this would have come up sometime before considering how important the bot is to some folks.

    Either that or a discussion about how the IP address is being used instead of of an actual site name. (As far as I know, Google doesn't have a problem with indexing an IP address.)

    I was hoping someone else would weigh in on this one as I would have liked to hear their take about this question. I'm glad though you got it working. Please let us know how things are down the road.

    edit: A link to the directory in question in a hopeful attempt to get Google to index it: http://69.162.104.186/gallery

    reedit: A suggestion: Rename gallery_index.html to index.html or at least copy it over. gallery_index.html isn't a known file name for Apache to pick up on as being the index for a directory.
     
    theapparatus, Oct 3, 2009 IP