Benefits and use of robots.txt file

I wnat to know what are the benefits and use of robots.txt file is. How can i use this file in my website.

Is it necessary or not for my website future.

 

Robots.txt file is basically used to instruct crawlers what files/folders to crawl or not. If you have some private information on your website that you do not want to share with others, you can simply block that file/folder using robots.txt.

You can simply create a robots.txt file and upload that in the root folder. you can set the instruction as required in your website.

 

thanks for the sharing known

 

using this file you can block you files and folders ..so that crawler can not read them.....

 

Let an app generate this file for you then upload it to the root of your site...simple
search google for robots.txt generator

 

I would recommend you to generate the robots.txt yourself as no application could configure what files or directories to be allowed and viceversa.