1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Which File Crawl By Crawle First

Discussion in 'Google' started by brad12, Mar 1, 2011.

0
  1. #1
    Hi I wanna one question, in you directory suppose robot.txt file is present. when search engine crawler crawl this file, I mean which file will be crawl by crawler first, is it index.html or robot.txt.
     
    brad12, Mar 1, 2011 IP
  2. FREE BET

    FREE BET Peon

    Messages:
    927
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #2
    it looks at the robot file to see if it has any restrictions...
     
    FREE BET, Mar 1, 2011 IP
  3. luken

    luken Member

    Messages:
    74
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    41
    #3
    I thinks always site-map crawled First....
     
    luken, Mar 1, 2011 IP
  4. SureShopApparel

    SureShopApparel Peon

    Messages:
    15
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Do you have to direct the crawler in your webmaster??
     
    SureShopApparel, Mar 1, 2011 IP
  5. bogs

    bogs Active Member

    Messages:
    2,142
    Likes Received:
    16
    Best Answers:
    0
    Trophy Points:
    80
    #5
    not sure about that, since its kinda part of their algorithm. And how exactly they perform the crawling only google knows.
     
    bogs, Mar 1, 2011 IP
  6. WishBone

    WishBone Peon

    Messages:
    2,566
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #6
    I think robots.txt are now getting crawled but it is one of the first file bots looking to see which pages are restricted or not set to be crawled.
     
    WishBone, Mar 1, 2011 IP
  7. Surajdeshmukh

    Surajdeshmukh Peon

    Messages:
    106
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    I agree with you.
     
    Surajdeshmukh, Mar 2, 2011 IP
  8. rwsseo

    rwsseo Peon

    Messages:
    546
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #8
    IMO crawler first crawl robots.txt file and then another file.
     
    rwsseo, Mar 3, 2011 IP
  9. DoDo Me

    DoDo Me Peon

    Messages:
    2,257
    Likes Received:
    27
    Best Answers:
    0
    Trophy Points:
    0
    #9
    robots.txt first. A decent bot will first check it and also regularly check this file.
     
    DoDo Me, Mar 3, 2011 IP