hi, I am using hardware load balancer. my wesite name poiting to the loadbalancer ipaddress. In google when i give site:domain.com it return all my indexed page. when i give site:mydomain ipaddress (ie load balancr ipaddress) its also return the number of pages. Is anything did i wrong as gogole indexed like this? domainname.com/1/xyz.html domainname's ip address/1/xyz.html both are going to same page. Please help me in this issus. Thanks in advance
I was also met this issue by using CDN etc anyway method 1: place robots.txt on any place you do not want to get indexed, you might need a dynamic robots.txt created by current host name method 2: use absolute path with full domain name on link herf. method 3: use canonical meta (may not work if your site is very very dynamic)
Can I disallow the IP addresss in which my ipaddress is hosted to avoid indexing in Robots.txt? If I disaallow, will it not create any problem since my website is hosted in that IP?