Importance of Robots.txt file

guys

does Robots.txt file affect your SE ranking.....

i am not using Robots.txt file in one of my website....

please tell me is there any harm?

 

Hi Friends in my opinion Robots.txt file will surely affect SE ranking because When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots.txt file is a simple text file (no HTML), that must be placed in your root directory



Dear friend in short it will surely affect SE ranking.

Regards

Sarah

 

It is advisable to keep one. The robots.txt patterns are matched by simple substring comparisons. Care should be taken to make sure that patterns matching directories have the final '/' character appended, otherwise all files with names starting with that substring will match, rather than just those in the directory intended.


source:wiki

 

my i am running a site without robots.txt file and it is working very well on SEs

 

Hi Friend,
I think it's give some help As SERP point of view......
Thanks!

 

Robots.txt does not affect your search rankings, Google (and other spiders) will find the links anyway. Robots.txt is used only to disallow certain robots from indexing your site or viewing pages with your site, as an example you would not want your "thank you" page indexed so you can block it from robots.txt

Of course not all spiders are legitimate and will index your pages anyway, plus anyone who know about the robots file will search there first anyway so you will be showing these people what you don't want them to see.

Kill robots.txt and add a meta tag to your headers instead

 

With robots.txt we can allow or disallow which file that should be crawl..so do not much eat the source of your server..and do not affect the SE

 

ITs not effecting any SE rank, this only to avoid crawl some pages,dir,file etc from Google crawl

 

In the earlier days, using robots.txt was one way of ensuring your site did not get the duplicate content penalty, especially if two or more pages were similar in most respects.

If you want to disallow certain pages, use it.

The only way it could affect your ranking would be for you to ask G,Y,MSN not to follow those pages which could possibly get you a rank

 

Although the robots.txt file is a very important file if you want to have a good ranking on search engines, many Web sites don't offer this file.
If your Web site doesn't have a robots.txt file yet, read on to learn how to create one

 

Eh? Lol 0/3 nice one dude

Want to see just how bad it can get?

http://www.whitehouse.gov/robots.txt

Code (markup):

 

robots.txt file prevents pages for search engine crawling.

 

robots.txt file prevents pages for search engine crawling.

 

you can use a robots.txt file to list your sitemap. either sitemap.txt or sitemap.xml

 

on default... all the pages on your site are allowed... you need robots only if you want to disallow the search engine to crawl the site.

 

yes.google robots will give more effect with ur serp.
more over your setting of robots texts

 

Cool, that might be the longest robots.txt file!!!!

 

It will help with the duplicate penalty you can get for having/ not having www

 

Does not affect search rankings using of Robots.txt file. it is helps easy to crawling spider.

 

As of now, robots.txt won't affect you ranking and you won't need one unless you want to block spiders from crawling your entire site.

But it is advisable that you have at least a black robots.txt file uploaded to reduce the number of 404 hits caused by the spiders. Check your server stats.
(This won't affect you if you are running a wordpress free hosted blog about Indian festival 2009. )

-Neo