Hello! i have a few pages that I don't want to have listed in google, but I want the mediabot to drop by because there are adsense ads on the page. How will I do this with a robots file?
To tell Google not to spider the files mypage1.html, you would write User-Agent: Googlebot Disallow: /mypage1.html if im not mistaking.
Well since the Adsense bot and googlebot are different you can just disallow googlebot from your folder. So something like... User-agent: GoogleBot Disallow: /mystash/ Code (markup): ...should work. Ross