This is my directory layout: www.mysite.com --> html --> file1.html www.mysite.com --> html --> file2.html www.mysite.com --> html --> file3.html www.mysite.com --> images--> img1.gif www.mysite.com --> images--> img2.gif www.mysite.com --> images--> img3.gif www.mysite.com --> button --> button1.gif www.mysite.com --> button --> button2.gif How would I rescrict access from all bots to (file3.html, and the images folder, and the button folder)? I've already tried something but I want to make sure it is right. Thanks.
Would this work? Robots.txt User-agent: * Disallow: /images/ User-agent: * Disallow: /button/ User-agent: * Disallow: /html/file3.html Code (markup):
Yes, this will work. There is an official robots.txt file generate in Google Analytics which allows you to generate your own robots.txt file in a user friendly interface.
This is the general format of robots.txt: User-agent: * Disallow: /images/ Disallow: /button/ Disallow: /html/file3.html