Hi, Yes, it is important. It allows certain Search Engines to index your site. And helps you go on in SEO.
It is important. Only Robots.txt makes the search engines to crawl your site and display in the search results.
with robots.txt you tell search engines what pages/files their crawl robots can visit. see more info on http://www.robotstxt.org/
Yes, you should definitely have one. Also disallow search engines from indexing your cgi-bin and images folders.
It may be important. It tells the search engines which pages on your domain to crawl. If you are like me, you want everything crawled regardless, so you can set it to all or just not use it. If you have a normal website, there is no reason to not crawl everything. Google is smart enough to know what to index and put in SERPs. Google will try and look for a robots.txt. And if it does not find it, it will show as a 404 not found error in your reports. No harm at all. If you hate the errors, then add one. Stephen C
I agree with everyone else robots.txt is important cause it lets you dictate which spiders can access your site and which part of the site. Also robots.txt allow the spiders to access your sitemap.xml. This can be very important to getting all of your URLs indexed.
It is not that important if you dont have something called robots.txt in the first place. If you do have, then it becomes important if you have what kind of rules in place in that file. You should take care not to prevent the indexing of certain pages you actually want indexed.
It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page, you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty. Also, if you happen to have sensitive data on your site that you do not want the world to see, you will also prefer that search engines do not index these pages. Additionally, if you want to save some bandwidth by excluding images, stylesheets and javascript from indexing, you also need a way to tell spiders to keep away from these items. So I think it is very important ...
yes Google Robots.txt importance for website and in SEO bcz if you put sitemap link in robots.txt so google cache regularly each and every page.
It has nothing to do with sitemap.xml. I personally don't use any robots.txt rules, just nofollow on the right links. I just download the default file from Webmaster Tools and set it up so Google feels happy.
Check this link it will give more information about robots.txt http://www.webconfs.com/what-is-robots-txt-article-12.php
Hi Dear Yes, it is important. It allows or Disallow certain Search Engines to index your site. and help for seo prospective. Thanks
I think if you are a search engine optimizer then it is of most importance to you..But from a user's perspective it is of no use..because in general we don't bother about who to allow who not..It is for SEO niche..and he/she only can understand it's importance.. I believe we should have robots.txt in our root folder..