block all sub domains

Hello

How to confgiure the robots.txt to exclude all sub domains from engines

ex. *.domain.com

 

hi Nick,

How is the subdomain set up in your website? Is it part of the same web space as the main domain, where a request for sub.domain.com ends up pulling content from a /sub/ subdirectory tied to your main hosting space?

The same would be true if the subdomain has its own unique hosting space. Simply place a robots.txt at the root level for the subdomain and the spiders will take care of the rest.

If that's the case you should be able to simply place a robots.txt file inside this subdirectory and have it applied only to the subdomain. the search engine spiders will request a robots.txt file from each unique domain, and they see subdomains as unique domains.

If you're doing any kind of 301 redirect for the subdomain you may need to do something a bit trickier. For example if your server is set up forward requests for sub.domain.com to sub.domain.com/sub/ via a 301 redirect it can end up messing up robots.txt for the subdomain if you don't give the robots.txt some special abilities. Or set up the redirect to be a transparent redirect.

It should be fairly easy to do in either case, but how to do it may differ depending upon your exact setup, what type of server you're on (*nix or IIS), etc. As a general rule all you need to do is get the robots.txt to sit at the root level of the subdomain address.

 

User-agent: *
Disallow: /sub/

 

You can not restrict each subdomain with the main robots.txt as Google considers each subdomain as a new website and Google crawlers look for the robots.txt for that subdomain to check any restriction. So if you want to block all of your subdomains, you will have to create separate robots.txt for each and every subdomain and write following instructions there:

User-agent: *
Disallow: /

By adding this code in each subdomain's robots.txt, the sub-domain would be skipped by crawlers and they would not get indexed.