Is robots.txt file necessary?

Hi
Is it necessary to put robots.txt file in the root directory of website, i know it is helpful to control crawlers, but if i not put this file in the website, does this cause some problem, please advice
Thanks for your support

 

It isn't necessary yet highly recommended although there are tons of sites that rank without it. Without it all pages are do-follow and index.

-chris

 

robot.txt file is not necessary until you don't want to make any restriction for search engine

 

Only if you want to block parts of your site from the search engines.

 

no it is not necessary

 

good question mate

 

Yes if you are not using robot.txt file then crawler will automatically crawl all the pages of your website. It is necessary if you want to restrict any particular page to be indexed.

 

Do most popular website builders like joomla and drupal auto include the robot file? I am hoping so.

 

It is not necessary. If you want block some pages in search engines, it helps to you.

 

You need to add robots.txt file to some crucial pages which should not be allowed to be crawled by SE like Contact Form page.

 

robots is used to block some part of the website you don't wish search engine too index it . for example client's personal detail etc

 

Robots.txt file is not necessary to use every website, if you want to block specific pages or any specific directory then you should use robots.txt file on root server.

 

its not necessary, but Google recommends doing in their webmaster guidelines, so why would you not?

 

The robots.txt file is a great way to tell spiders where your sitemap is located.

 

robot.txt file is not necessary

 

Really Robot.txt file is not necessary to the website. It is used to restrict the search engine bots to crowl any parts of the website.
If you not want any restriction to search engine crawling then no use of robot.txt

 

it is not necessary to put robots.txt. it is just a file that tell the search engines about some special commands

 

Definition
Robots.txt is a simple text file without any HTML code in it and it is a good practice to place it in the root directory

Purpose:
When ever a web spider or crawler comes to your site for the purpose of crawling then it first of all search out robots.txt. From this file the spider gets this information about which pages of your site to be indexed and which to ignore.

Contents of Robots.txt file:
As its a text file so it can be created in any text editor. Robots.txt consist of records which contain information for search engines. Each record consist of two fields the user agent line and one or more of the following directives;
Disallow
Allow
$wild card support
*wile card support
site map location

 

No, it is not mandatory that you have to include robots.txt file in the root directory of your website. Basically, this file put restrictions some of your web pages of website from being crawl by search engine. If you don't want to restrict some of your web pages from crawling, then it is not required to include this file.

 

You don't need it but even if you do not intend to restrict bots you can use it to define the location to your site map like:
User-Agent: *
Allow: /
Sitemap: http://www . yoursitename .com/sitemap. xml