Hi, I would like to know how to block a particular folder from search engine crawler. I just want to block one folder only and not the entire website. for eg. www.website.com/folder-to-block/index.htm now how to disallow this "/folder-to-block/" folder by using using robot.txt? Thanks
you can use this code to block search engine to crawl the folder User-agent: * Disallow: /folder-to-block/
Create a robots.txt file in your server and put the code User-agent: * Disallow: /folder-name This will remove the folder from the indexing..
If we can define Robot.txt file on webmaster tool it will be easy to index your WebPages .In robot text file we need to define which file of folder need not to be indexed.
Also, you can make sure that the folder/page is not accessible/linked from any of the pages from the site other than configuring robots.txt.
But this way doesn't work if the folder is indexed yet. If Google indexed this folder, the only way is defining it on robots.txt using Google webmaster tools or any other way, then when Google reads the robots.txt in next times, it will not indexed the pages on that folder more.
As the others are saying you could place a robots statement in the meta (example shown) in the header if the document is html. <html> <head> <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW, NOARCHIVE"> </head> Code (markup): If the content in question is a pdf, doc, txt or some other exotic file/doc type you can create a separate robots txt file on your site name.com/robots.txt or add a no index to your .htaccess (if you have one.) ROOFIS