As an amature in building my website I learned the hard way not to put the robots.txt file in my main directory (public_html) where all my web pages are stored especially if I have all my download links or members area links (that I am selling) in the robots.txt file because people can easily find it. It is better to put is elsewhere.
Is this a joke? And how do you think search engine spiders will be able to find it from the sub directory? I suggest you move robots.txt to a flash drive and hide in your basement.
Firstly it depends on the configuration of the server about directories!!!! Also if the permissions are set to allow to read to the file, spiders are clever enough to find it and find all the info they need. I think there is a high possibility that the crawlers will assume you don't have any robots.txt file if they can't find it easily. The level for easiness depends on the company's crawler. But some spiders are "lazy" and if they have to "digg in" to a site to find the robots.txt they probably leave the site for a later crawling or even abandon it completely... (if after several attemps it is not easily found) Again it depends on the company's crawler. Yahoo has it's own , Google another and so on... I don't understand why anyone would like to move robots.txt to a non typical directory.
Buddy, what are you saying ? robots.txt must in the root directory of a domain/subdomain. It should be like this http://www.yourdomain.com/robots.txt . If you put the robots text to any other directory rather than in root directory, spider ignore the files.