Hi all...... As it is known to all that robots.txt file is use to specify how search engines should crawl your site's content.....but how much it is important to have it? Does it help in SEO?
if you ask me... i never used it or even edited it... but yeah... using robots.txt you can do many things.. like... you can even stop google to crawl your specific content on webpage! for more info on robots.txt you can search DP
It can be very important to block spiders from crawling directories on your site. Esp if you're using a CMS
You use a robots.txt if you want a SE NOT to crawl a certain section usually. They are not really that important for any other reason, I personally don't bother with one.
Yes its help in seo. I can disallow spider to crawl some of your pages (example. Database and other sensitive pages which don't you want explore).
It's not important unless you are trying to block access. If all your content is open to the public, robots.txt doesn't magically score you any points.