Hi, For the purposes of SEO, I want to block out terms and conditions using robots txt. Since terms and conditions are important for quality score, will blocking such pages with robots txt stop the AdWords spider from seeing that we have a terms and conditions page? Thanks in advance.
There would not be any problem like violiting google terms and condition of service by blocking your term and condition of services via robots.txt Since still there would be the link for it on your website for human visitior. You are just trying to block so that search engine could not read it. Cheers
That's an old article. Nowadays, the rules have changed. There is no need to exclude all 3 major crawlers in robots.txt. You can read more about it on the AdWords Help here
1. Use GS generator software. 2. Update your robots.txt file. 3. Add your site to Google Webmaster Tools. 4. Submit the address of your GS file. A good first step in getting your webpages indexed is to have a GS. And with automated GS generator software, it improves the possibility of your webpages being indexed and showing up in search engine results. Cusimano.Com Corporation ( http://www.c3scripts.com/ ) is the developer of the sitemap.pl Google Sitemap generator software and other cgi-bin software scripts to help affiliates increase their affiliate marketing commissions.