Ok, i have a script on my site, it is a wap site script, it has forums, chat, blogs, etc all for wap phones or wap enabled emulators or browsers. It is at my site with a wap subdomain and I always wanted to know, because there is alot of posts and stuff, (well there was), but you cannot see posts and topics unless you are logged in. Is there anyway to give robots.txt a username and password that it can use to login and search the site for posts etc like a normal forum. Thanks
No, you can't. You need to open this section to the public or publish a small abstract of its content.
ok, i was thinking to alter the script to add like a extra page where you can view topics without being logged in and link it from main page so that they can index them, well I thought i would try it the easy way, lol, but thanks anyway
If you forum is vbulletin there are mods to allow this on vbulletin.org. I think it's a bad idea though, as this I believe falls under "cloaking". You should let the spiders have the same permissions as unregistered guests. -Raymond
Why would you want your content indexed by the robots, which would be readable via cache yet want your board private?
Its the way the script is made, it only allows registered to view things. I have made a section for public to browse everything anyway,so thanks for that anyway,