Since robots.txt resides in the root directory of a website, all you have to do is to download it by typing in the url to it. Isn't that a security threat?
In order to work, robots.txt needs to be world-readable. If you can't download it from http://root.tld/robots.txt, then a robot can't either. If you have something to hide, your robots.txt file is not the best way to conceal it. For example, whitehouse.gov/robots.txt was once famous for its size and some of its curious entries. If you see robots.txt as a "security threat", you're probably using it wrong. If you don't want people and robots to see certain pages, you should protect the pages in your .htaccess file. The robots.txt file is not a security system.
It all depends what you are hiding through robots.txt. If you are hiding some confidential information through then it is a security threat? so better use htaccess or password protected pages to hide confidential information on the website.
using robots.txt, you can provide security to your pages for crawlers then after bots will not crawle your some confidential pages.
The robots.txt file is not a security system. But i don't know how to do, Xenlightenment-Teisho says that "you should protect the pages in your .htaccess file" detail????
Someone could see the robots.txt but they cant upload a new one unless they have hacked your system which by then its to late. robots.txt only blocks SE spiders from seeing the page. You can still navigate to it by typing in the url.
Yes, but how can I change the robots.txt file. Since FTP is not. Can someone give me some details, too? I want something out there to the public but do not know how to edit this file ___________________________________________________________ piese auto online auto dealers
I agree with manish.chauhan here, robots.txt is NOT the the absolute best method for stopping bots. To be sure, use htaccess or password protected pages. Also, at a lower effectiveness level, nofollow is now not paid attention to by search engines like Google. The search engines decided to be the ones who decide what they crawl, not the webmasters. PageRank sculpting is fairly redundant now. Just my 2 pence!