Hello, I have a robots.txt file that has the following content: User-Agent: * Disallow: Disallow: /developmentsite/ Code (markup): I'm looking to stop the indexing of the development section of my site (anything that falls after /developmentsite/. The robots.txt is placed in the root of the site. When I check it using http://tools.seobook.com/robots-txt/analyzer/ it says that robots are allowed to index this area. I'm very confused and would appreciate any help you can give me. Best wishes, Graeme
Are you sure it says that? I just tested it: Url: http://www.test.com/developmentsite/ Robots disallowed: All robots Code (markup): Thats a very, very poor URL for your development site.. It is completely up to the crawler whether it honours your robots.txt rules. May not be a problem with Google but it could become indexed elsewhere. Your development site should ideally be offline, if thats not possible why not create a sub domain? dev.yoursite.com. If it must be online make sure it is password protected with .htaccess rules so that it can't be accessed at all via the outside world.