I work for a web host but for some reason I am really struggling with a question a client brought up to me. I'm not sure of the best way to handle it. The problem is that Google is listing one of his custom nameservers for his domain and indexing it. It's also indexing the domain those nameservers are assigned too. The domains are indeed different between the custom nameserver and his domain. I'd rather not give them for client privacy concerns if possible. Example: Client has (2) nameservers like: ns1.coolthings.com ns2.mystuff.com Notice both custom nameservers do not share the same domain, not like it matters though. These are assigned to strangestuff.com for DNS. Google is indexing the following: http://ns1.coolthings.com and http://strangestuff.com This is because of the cPanel and Apache default behavior to display the website being shared by that child name server IP address. The client wants to prevent the duplicate listing but does not want the down time of changing that nameserver to another IP on their block. No way to use robots.txt into this because the nameserver has no document root. I was thinking that a cPanel/Apache vHost entry could be built with a document root and then insert a robots.txt disallow * inside of that document root. I'm not too sure how to build that or if cPanel allows, but it's my best thing to go on as of yet. Any suggestions? Regards,
OK, so from what you said, the problem is thus: client has a domain (dom1) and the two name servers (ns1 and ns2). ns1 and ns2 are obviously handling DNS for dom1. The problem is that Google has indexed dom1 AND ns1, and ns1 is displaying the same content thus he has a duplicate content problem. If ns1 and dom1 are on the same IP, the best way would be, like you said, to setup a vHost entry for the ns1 domain and then put a robots.txt file there (or a 301 redirect to send all traffic to dom1). I've never really used the admin panel of cPanel before, but it should be straight-forward to setup a vHost and disable everything else except for the document root. Otherwise you could simply create the Apache vhost conf file manually, but I don't know how cPanel would like that (i prefer to do these things manually in any case)
You could create the cpanel entry then manually change the document root in httpd.conf. Remember to use the cPanel distiller or your changes to httpd.conf will be overwritten when upcp is run (usually every night). http://www.cpanel.net/documentation/easyapache/customdirectives.html
Hi, The only way to really prevent this is to make sure httpd is not listening on the IP address that is being used as a name-server. The interesting question is why it got indexed as usually google will only index sites that have links to them. searching for the following in google might explain this: link:ns1.coolthings.com edit: You could also just add ns1/ns2 as a sub-domain and add a robots.txt. This does not require you to edit the httpd.conf at all.