Hi, I am new to this topic. Can anyone please tell me where it's used ? it's importance ? etc... Thanks
You have to upload it on your web host Read more here http://www.robotstxt.org/ I upload on my host today only to allow all search engines to spider my site.
it can tell search engine spiders where to index files and other things related to that manage which can etc
I personal don't like the old /robot.txt because if someone was trying to deface your website it will help them out a bit but hey thats my opinion.
start visiting http://www.robotstxt.org/ . it's small text file which tell useful information about your site to search engines