What is robots.txt? and can i use this in my blog how? plz any body help me. my blog adderss is musicnewspk.blogspot.com
get more info about robots.txt. It's called robots exclusion standard and can be used to control which pages will be indexed by search engine bots.
Robots.txt is just a regular text file but by defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site... Here's how to create your own robots.txt file.
Robots.txt is a text file that is used to control spiders that visit your website. This file grants access to certain folders, file types, and specific files depending on the robot accessing the site.Its actually a file which sits in the root of a site and tells search engines which files not to crawl.