According to me here i define one link,may be it's useful for you.Check the following link. http://www.searchtools.com/robots/robots-txt.html
I think this link clear all your doubts related robots.txt file http://en.wikipedia.org/wiki/Robots_exclusion_standard
If u want to know more about what above line is used in robots.txt then u need to know what is robots.txt file and why we use it.
When, there are any error in our website page or we want to remove the page from our website then we use robot txt.