hello, I want google to index mysite.com/folder/ But i dont want any files/folders in it to be indexed. For example, I dont want google to index any of the following. mysite.com/folder/1.htm mysite.com/folder/2.php mysite.com/folder/2.html mysite.com/folder/somefolder1/ mysite.com/folder/somefolder2/ mysite.com/folder/somefolder3/ how to make google to index only mysite.com/folder/ ? ============= If I use mysite.com/folder/index.php instead of mysite.com/folder/ and use User-agent: * Allow:/folder/index.php and User-agent: * Disallow: /folder/* This can solve the problem. But i dont want to use mysite.com/folder/index.php So, is there any other solution ? Thanks.
robots.txt is not so powerful than before.. sometimes, you cant control it... And, the page link juice will disappear... So, you can use a meta tag <META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW"> for those pages which you don't want goole index.. Best & Ideal robots.txt: User-agent: * Disallow: Sitemap: http://www.yoursite.com/sitemap.xml sorry for my bad english!
as written here, you can use the meta tag, you can also use the webmaster tools and you can use the robots, anyway, your solution doesn't look good to me, you better put all files in other folder.