Hi hello to all members of dis forum i have a question regarding robots.txt file can anyone plz tell me about robots.txt what is robots.txt and how we can create dis file n whts d use of dis file in SEO . n plz tell me about competitor analysis in seo.. Plz help me out Thnx 4 advance
You must know What is Robots.txt? Robots.txt is a text file which you have to put into your website main directory. When you add the Meta robots file the url will create like http://www.yourdomain.com/robots.txt this text file tell to crawler that which pages you allow to crawl and index in search engine or which pages disallow to crawl and index by search engine. If you have a blog then you can use Meta Robots SEO WordPress Plugin Cheers,
Hi search engines frequently visit your site and index your content One way to tell search engines which files and folders on your Web site to avoid is with the use of the Robots.txt
using robots.txt file you can restrict the spider by crawling some important page of your website. It is easy to create robots.txt use notepad and save it as robots.txt. you can also create using google webmaster tools.
it's not necessary to have it , but its good to use it as you tell most crawlers what is allowed for crawling