What is Robo.txt

hi frds can any one tell me wat is robo.txt in web site ..and how it helps to us..??

i know its a silly but pls give me your inputs

Thanks in advance

 

oppps you posted in the wrong sub forum please ask a mod to move it using the report button before someone else does and you get an infraction point.


Have a read of http://www.robotstxt.org/ for the whys and hows.

basically it just tells search engine bots what ones are allowed access and to what pages.

 

robots.txt are used to control spiders coming to website.. you can get more details at robotstxt.org

 

Robot.txt can control the access of search engines like google and yahoo, bing,
search engine will still crawl the pages but will not index them.

 

robots.txt file controls which files and folder bot can see.

For example:
create notepad file name robots.txt and put

User-agent: *
Disallow:
Disallow: /cgi-bin/
Sitemap: http://www.yourdomain.com/sitemap.xml

 

robot.txt is a file in the website that controls the search engine robots to crawl over your site. It restrict the crawler, the links or the portion of the site which you don't want others to view. It is somewhat a controlling power on your hand.

Sunshine Coast Internet Marketing

 

robots.txt can be used as a set of simple commands towards the Search Engine spiders. It's often used to tell them not to crawl certain directories on a website - for example admin areas.

 

Sorry , it's robots.txt

 

robo.txt is spider .which is used by search engines to check the quality of the websites.Search engine lounches
it's spider to crawl websites and give them ranking according to quality and backlinks