HI All, I am not clear much about for robot.txt, i just have an idea about it that by using robot.txt file we can stop the indexing of pages for any search engine. But i think if your site is strong in search engines, i mean if many pages are indexed in search engine then search engine like that type of site. so why we stop the indexing of pages in search engine?
You might have login pages or pages which can be accesed by more than one URL and want to block them from search engines.
The robot.txt file help give search engines directions and limitations when search engine bots and spiders visit a website. Some websites have pages and directories that search engines should not visit, crawl, and index. The robot.txt files allows webmasters to define these pages as well as the pages that should be crawled.
Not only that but their presence helps keeps your server's error logs nice and squeaky clean. Nothing worse than trying to find a broken link only to see thousands upon thousands of them all referring to /robots.txt
in addition to that you can also place the location of your sitemap on the robots.txt and it will help crawlers to index the right pages sitemap:urllist.txt
Can you tell me practically how we do it i mean what type of setting we have to do in robot.txt file for stopping particular url................... ???