hey, why robots.txt file is important? what is the mean of- User-agent: * Allow: / and User-agent: * disallow: / I'm really so much confused! your answer will be highly appreciated! thanks,
Robots.txt file is useful for search engines like google, yahoo, bing etc. That file was created to avoid lots of websites without no content or testing sites to be indexed on search engines. And that file will be automatically allow indexing your website after some specific time, to verify that your site is original lol This answer may help you
It's import for SEO. I use it to block all spiders and crawlers when I have a project online while it's in the development stage. You can also use it to block content you don't want appearing in search results, for example if I have a captcha folder for storing the relevant php scripts or an includes folder for storing all the reusable components on a website I'd use the following commands. Disallow: /captcha Disallow: /includes
sorry! I'm not so clear... If I write- User-agent: * Allow: / -that means all contents are open for search engines? and suppose I don't want to block anything then what will be the task for this robot?
Yes. User-agent: * Allow: / Means that ALL search engines are allowed to crawl and index ALL your site's content User-agent: * disallow: / Means that NO search engines are allowed crawl and index AN of your site's content If you want everything to be indexed you can use the first command or omit the robots.txt file altogether. Note: Crawlers and Spiders can ignore the commands in the robots.txt file.
@jjosephs This is really informative, I guess robots.txt is kind of a control the web developer has for making the search engines crawl the pages, the developer wants. tnx