Webmasters wanting to exclude search engine robots from certain parts of their site often choose the use of a robot.txt file on the root of the server. This file basicly tells the bot which directories are supposed to be off-limits. An attacker can easily obtain that information by very simply opening that plain text file in his browser. Webmasters should *never* rely on this for real security issues. Google helps the attacker by allowing a search for the "disallow" keyword. The Google Hacking Database (GHDB) appears courtesy of the Google Hacking community. what do you think?
It's not a Hacking tool per se, it can just make a hack a lot easier if you have a file telling you where you're not supposed to go.
If you are relying on robots.txt file to keep people out of places they shouldn't, you have much bigger problems.
It is nice to know DP have good music taste. No i do not trust robot text to keep me out i just saw it when i scan my files for security. So i guess i can keep it the hackers will find the files anyway
if you have a cpanel and many domains in the same root stuff happen to come there but that has noting to do with robot.txt
and he got banned ... o well not it does not the good once still follow the rules. Google follow the robot.txt i guess so save loads of bandwith not to allow them to some places.
no robot.txt is not a hacking tool .it is a file associated with the server, which stores the information such as meta data about an website.so that when a query is send to the server,it checks the robot.txt and produce the result with in a small period of time.
I think I agree with Digitalpoint... Hackers can and likely do use Robot.txt to hack sites, but only sites from people that are not properly using it. A bot can not access a page/file they do not otherwise have access too, so by disallowing ANYONE but you into certain areas of your server also disallows bots. Another key is to not link to areas bots should not be getting into...
all you need is simple .htaccess code to prevent people from even looking at robots.txt to begin with. If the ip addresses do not match the ones that the bot use, send them to another page.
You can also use this to your advantage; Put a fake "admin" folder in robots.txt and create a fake looking yoursite.com/admin -- complete with login and all but just they won't ever be able to log in (It is a fake..) -- Or you can log their details and ban them from the site, etc, whatever you want..