Explain The Whole Process Of When Google Crawls A Site

Discussion in 'Google' started by AlphanetChrisC, Nov 24, 2008.

  1. #1
    I've always wondered how this works.

    When google crawls, it clicks all the links on the site, then start crawling from the links? or how does it work?
     
    AlphanetChrisC, Nov 24, 2008 IP
  2. zk200083

    zk200083 Peon

    Messages:
    211
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #2
    It would begin from the top to the bottom, from the left to the right
     
    zk200083, Nov 25, 2008 IP
  3. Voorsie

    Voorsie Active Member

    Messages:
    115
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    53
    #3
    You say google clicks all the links on your site, that's true, but that's already a few steps after the starting of the process.

    First of all google needs to find your site. This can be via another link on an external website which google already knows, or by submitting your site to google itself (not recommended).

    When google knows your site, it will look to your robots.txt. The crawler can read there what to index, and what not.

    After that google is going to follow all your links (not the first time), and index them.

    Don't think your whole site will be indexed in one time, this can take some time.

    You can help (and maybe speed up) the process by uploading a sitemap to google at https://www.google.com/webmasters/tools/
     
    Voorsie, Nov 25, 2008 IP