I am operating a paid membership website. I have blocked all search engine crawlers from accessing my membership pages to ensure that it does not get indexed. Only my non member pages are indexed. I want to add adsence to my member pages though. However, my adsence codes are not updating on my website. It is not showing anything relevant to my pages. After a day I even tried setting the keywords manually and still nothing. It is like it is not communicating with those pages. How can I allow this to be updated by the adsence crawler without jeopardizing those pages to being indexed? (I don't want it indexed)
hi tech, Add the following to the top of your ROBOTS.TXT page: User-agent: Mediapartners-Google* Disallow: OR... use Section Targeting to control the ads that appear on your pages. You can learn more about section targeting by visiting the Adsense Help Section. Hope this helps.
Thank You! Does this look ok? Keeping in mind that I want to block the pages above from being indexed but allow adsence to update my codes.
Hi tech, I honestly can't say if it "looks" ok. I basically format my robots.txt pages the same way. They are all basically similar: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /css/ Disallow: /cgi-bin/ Disallow: /images/ User-agent: * Disallow: /*.gif$ Disallow: /*.jpg$ User-Agent: Whatever1 Disallow: / User-agent: Whatever2 Disallow: /
Thanks anyway. I know what you mean. I believe the robots txt file is more confusing than all the coding I did on my website. It seems to be doing the job so far. Unfortunately google indexed two pages that I did not want up, terms of use and policy, before I put this into place. i don't know how to remove it from google and its affecting my keywords also.