What is Robots.txt ?

In SEO Pointer view Robots.txt is just allow or disallow the search engine spider to index your website pages, folder, css, images etc by write a simple code in robots.txt. For example if you want to disallow search to index certain page then code is

Disallow: /services
Disallow: /services.php

Or your want to disallow folder then code is -:
Disallow: /search
Disallow: /catalogues

 

I hope everyone read your post! It seems like too many people are considering using robots.txt as a layer of security.

Everyone, please read up on robots.txt at http://www.robotstxt.org/robotstxt.html and http://www.wwwchecklist.com/list/robots/

 

Robot.txt is a text file in which we provide the information for search engines which pages we would like to index or not in the search engines.

 

Robot.txt is a file which is used to control the crawlers from crawling specific page which you do not want them to crawl.

 

Yep robot.txt file is only to tell search engines which page of your website search engine should visit and which page of your website search engine shouldn't visit

 

Well you block any page you dont want to index you will get an error in webmaster...

 

So why do you get an error in webmaster? Are we talking google webmaster tools?

 

Also if robot.txt doesn't enhance security, what is suggested?

 

Robots.txt is a text file we put in our site in root folder to tell search robots which pages we would like them not to visit.

 

Thanks for great information shared, robot.txt file would be indexed in every page on search engines.

 

In simple, robots.txt file is a text file containing a criteria for search engine crawler which files to be crawled or which not.
Of course, it plays important role in SEO.

 

Hi Sandegownc,

I am very much confused in sitemap.html and in sitemap.xml can you explain the difference between both of them.

 

Not all search engines seem to obey this file though. Nevertheless it helps the major bots know what to index and what not to.