Im really afraid to deal with robots.txt file. I have a web portal which deals with latest news, including future price predictions and many other informations regarding commodities. Our wesbsite was showing up good in Google News, but since the last year we are not showing up in GN. When I checked the webmaster tools for crawl errors I have found about 50K Article content errors in the news section. The News error presently shown up as "Article disproportionately short" refers to the Pages which shows future prices in tables and has nothing to do with news. Im planning to include a robots.txt file for Google news so that the news bot do not index those pages. The question is how do I write the robots.txt file just for Google News. I want to allow only the news section and block all other sections for Google News Bot and allow all other pages for remaining bots. This is what I have written, User-agent: Googlebot-News Allow: /news/ User-agent: * Allow: / Code (markup): Is this the right way to do it? Or could it bring any harm to other indexed pages?
If you want that Google doesn't crawl and index your page, just put meta tag in that particular page: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
Actually it is not practical to sort out each page that I dont want to index. The portal has millions of pages and growing 100's day by day. I just want the Google News to keep out of the sections that doesnt come under the news category.