Quick SEO question for SEO gurus

Discussion in 'Search Engine Optimization' started by andrej, Feb 20, 2008.

  1. #1
    I wonder if a Google bot starts scanning my site, does it scan all the files and folders in my public_html/mywebsite/ file? Or it scans only the files that are linked from my website?
     
    andrej, Feb 20, 2008 IP
  2. astup1didiot

    astup1didiot Notable Member

    Messages:
    5,926
    Likes Received:
    270
    Best Answers:
    0
    Trophy Points:
    280
    #2
    First, the crawlers only read your HTML rendered pages via your top level web directory, second 90% of the time Google finds pages from internal and external links to those web pages. However, Google can find a web page that isn't linked to at all, this does happen, but not that often. Proper use of robots.txt can prevent the pages you don't want indexed from being crawled and indexed.
     
    astup1didiot, Feb 20, 2008 IP
  3. andrej

    andrej Notable Member

    Messages:
    3,210
    Likes Received:
    130
    Best Answers:
    0
    Trophy Points:
    220
    #3
    OK, thanks for replying, SEO guru. :)
     
    andrej, Feb 20, 2008 IP
  4. Bompa

    Bompa Active Member

    Messages:
    461
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    58
    #4
    How does Google do that?
     
    Bompa, Feb 21, 2008 IP