Give Answer Please

create Robot.txt file is must for website? why?

 

Robots.txt files give a signal to crawlers that the page is to crawl or not to crawl and to follow or not to follow.

 

Learn to use search. This has been answered many times on the forum and there are a gasquillion pages about it to be found in Google.


Start here.

 

Robot.txt file tells the spider that which web pages should be indexed and which should not. Because sometimes the spider crawls your admin page and root directory folder and display it in search result pages.

 

it instruct spider to crawl your website

 

Robot.txt file will helps Google to index your web page

 

It is used to intimate the search engine to crawl or not crawl a particular page in the site.

 

Hi All,

robots.txt tells the search engine bot which pages to crawl and visit. Its very important for a website.

Hope this would help,
Sudarshan - Marathi Calendar