Which File Crawl By Crawle First

Hi I wanna one question, in you directory suppose robot.txt file is present. when search engine crawler crawl this file, I mean which file will be crawl by crawler first, is it index.html or robot.txt.

 

it looks at the robot file to see if it has any restrictions...

 

I thinks always site-map crawled First....

 

Do you have to direct the crawler in your webmaster??

 

not sure about that, since its kinda part of their algorithm. And how exactly they perform the crawling only google knows.

 

I think robots.txt are now getting crawled but it is one of the first file bots looking to see which pages are restricted or not set to be crawled.

 

I agree with you.

 

IMO crawler first crawl robots.txt file and then another file.

 

robots.txt first. A decent bot will first check it and also regularly check this file.