I have heard that it is used to provide security to the site. But don't know how.. I know about the use of robots.txt