it lets you tell search engine robots and any other bots in general what you want and don't want being accessed or touched. Bots that adhere and read the robots file will do as you tell them, while bad bots that don't read it will do as they want. Basically it lets you control what gets accessed and what you don't want them to touch.