Hello, Is it possible to create a robots.txt file that will exclude all Australian based search engines (ie, google.com.au, au.bing.com, au.yahoo.com, etc) from indexing my website? If so, how would I do that? Conversely, can you create a robots.txt file that ONLY allows Australian based search engines (such as the ones listed above) from indexing a site? And even more specifically, is it possible to create a robots.txt file that allows only search engines from Australia to index just one subdirectory of a site, while the rest of the site is open to search engines from any country? Cheers, Andy
Unfortunately, the answer to all of your questions is no and to my knowledge - but I could be wrong there - all the SEs that you mention use the same UA for crawling in any country so blocking it would stop them indexing your site alltogether.
Thanks very much, appreciate the response, although too bad it wasn't the answer I wanted! Cheers, Andy