Hi, I host multiple sites on my webserver. My main site, eg. www.maindomain.com, is at the root of my webspace. Then the other websites are contained in a folder, eg. www.maindomain.com/site1/, and they have their own dot.com domain pointer pointing to them, so www.maindomain.com/site1 is equivalent to www.mySite1.com. Given that I have this structure, and that I want to keep all these sites separate, do I create: 1. a robots.txt file at www.maindomain.com/robots.txt that excludes all the folders from those other sites 2. and have another robots.txt at www.mySite1.com/robots.txt if I have any further exclusions beyond that? Does this make sense? Because when crawlers search www.maindomain.com, I don't want them to index pages from those other websites, like www.maindomain.com/site1/files.php. I want those other pages only indexed for their respective domain names, like www.mySite1.com/files.php. Am I on the right track? Thanks a bunch.
Absolutely ! Another approach is to redirect (301 redirect) visitors asking for www .maindomain.com/site1/ to www .mySite1.com/. Jean-Luc
If my robots.txt file at the root (www.maindomain.com/robots.txt) blocks off all the folders, but the www.mySite1.com domain pointer goes to one of the folders, will that folder get indexed as www.mySite1.com/files...., even though the robots.txt disallowed access to them from www.maindomain.com/site1/ ?
Yes, www .mySite1.com/blahblah.html will be indexed and www .maindomain.com/site1/blahblah.html will not be indexed. Jean-Luc