IMO... it is under the Robot.txt hopefully it solves your question. Because Robot.txt controls spiders what are the accessible parts/data inside your site.
If you have password protected directory, spider cannot get in side the directory. If member's are is a hidden link that only members know about, you need to supply a link to that hidden are in a page that is not already hidden from the spider.
You can program it in to your system to specifically allow user agents that are crawlers to access without a username and password, but it has to be done explicitly, and in the code of the software. This is quite common and you can find a good programmer to do it for you if that's not your thing. If you want, I can build this functionality for you, send me a PM if interested.
That's the way to do it mate. I was just wondering. Normally people have password protected areas to keep the content contained within away from public eye. If you allow user agents/bots then you are letting it out to public domain.
I can see where this would be true on a warez forum or porn forum or something, but most of the situations ive seen, you want to hide member functionality from people, not the actual content they generate. My adword campaigns are useless because all I have is a landing page. If the spider could crawl my user generated content, my quality score would fly up.
This is the primary reason people do it, if you give a login and username to a site, or you to filtering by user agent, you can allow spiders to crawl inside your site, giving you serps and relevant adsense results based on your content, yet still keep it private. For an example of good usage, check out experts exchange forum threads: http://www.google.com/search?hl=en&q=site:experts-exchange.com/Miscellaneous/&btnG=Search Google crawls it, and puts the page in serps, yet you can't really read most of what's posted unless you log in. (By the way, experts exchange is a great site for those hard to find problems, their experts are great).
If you login to your webmaster central in google,You can find a code that you can place anywhere you do or don't want google to crawl I don't have it handle to give it to you.