1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Is robot.txt a Hacking tool?

Discussion in 'Security' started by TheSyndicate, Jul 16, 2009.

  1. #1
    Webmasters wanting to exclude search engine robots from certain parts of their site often choose the use of a robot.txt file on the root of the server. This file basicly tells the bot which directories are supposed to be off-limits. An attacker can easily obtain that information by very simply opening that plain text file in his browser. Webmasters should *never* rely on this for real security issues. Google helps the attacker by allowing a search for the "disallow" keyword.

    The Google Hacking Database (GHDB) appears courtesy of the Google Hacking community.

    what do you think?
    TheSyndicate, Jul 16, 2009 IP
  2. Dixie

    Dixie Guest

    Messages:
    46
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #2
    It's not a Hacking tool per se, it can just make a hack a lot easier if you have a file telling you where you're not supposed to go. ;)
    Dixie, Jul 16, 2009 IP
  3. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,363
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    315
    #3
    so should you use it or not
    TheSyndicate, Jul 16, 2009 IP
  4. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    36,127
    Likes Received:
    2,245
    Best Answers:
    389
    Trophy Points:
    710
    Digital Goods:
    27
    #4
    If you are relying on robots.txt file to keep people out of places they shouldn't, you have much bigger problems.
    digitalpoint, Jul 16, 2009 IP
  5. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,363
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    315
    #5
    It is nice to know DP have good music taste. No i do not trust robot text to keep me out i just saw it when i scan my files for security. So i guess i can keep it the hackers will find the files anyway
    TheSyndicate, Jul 17, 2009 IP
  6. justdoit1

    justdoit1 Peon

    Messages:
    100
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #6
    An important thing is that you should never place sensitive data inside the web root directory.
    justdoit1, Jul 17, 2009 IP
  7. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,363
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    315
    #7
    if you have a cpanel and many domains in the same root stuff happen to come there but that has noting to do with robot.txt
    TheSyndicate, Jul 17, 2009 IP
  8. kunnusingh

    kunnusingh Active Member

    Messages:
    563
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    70
    #8
    Use permission. Set to ony excute so people cant read your robots.txt
    kunnusingh, Jul 20, 2009 IP
  9. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,363
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    315
    #9
    Use permission. Set to ony excute so people cant read your robots.txt

    what do you mean with this
    TheSyndicate, Jul 20, 2009 IP
  10. Dixie

    Dixie Guest

    Messages:
    46
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #10
    ....Does that not make robots.txt pointless?
    Dixie, Jul 20, 2009 IP
  11. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,363
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    315
    #11
    and he got banned ... o well not it does not the good once still follow the rules. Google follow the robot.txt i guess so save loads of bandwith not to allow them to some places.
    TheSyndicate, Jul 20, 2009 IP
  12. thewebhostingdir

    thewebhostingdir Well-Known Member

    Messages:
    3,797
    Likes Received:
    82
    Best Answers:
    0
    Trophy Points:
    100
    #12
    "robots.txt" is a file which is used by web site developers to give instructions about their site to robots. It is known as "The Robots Exclusion Protocol".
    thewebhostingdir, Jul 20, 2009 IP
  13. anuradhan

    anuradhan Member

    Messages:
    365
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    40
    #13
    no robot.txt is not a hacking tool :cool:.it is a file associated with the server, which stores the information such as meta data about an website.so that when a query is send to the server,it checks the robot.txt and produce the result with in a small period of time.
    anuradhan, Jul 20, 2009 IP
  14. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,363
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    315
    #14
    well the hackers can see, well maybe they can see anyway what ever .-.
    TheSyndicate, Aug 7, 2009 IP
  15. Qryztufre

    Qryztufre Prominent Member

    Messages:
    6,072
    Likes Received:
    491
    Best Answers:
    0
    Trophy Points:
    300
    #15
    I think I agree with Digitalpoint...

    Hackers can and likely do use Robot.txt to hack sites, but only sites from people that are not properly using it. A bot can not access a page/file they do not otherwise have access too, so by disallowing ANYONE but you into certain areas of your server also disallows bots. Another key is to not link to areas bots should not be getting into...
    Qryztufre, Aug 7, 2009 IP
  16. scylla

    scylla Well-Known Member

    Messages:
    1,005
    Likes Received:
    30
    Best Answers:
    1
    Trophy Points:
    175
    #16
    all you need is simple .htaccess code to prevent people from even looking at robots.txt to begin with.

    If the ip addresses do not match the ones that the bot use, send them to another page.
    Last edited: Aug 7, 2009
    scylla, Aug 7, 2009 IP
  17. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,363
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    315
    #17
    could you give me a example what you mean?
    TheSyndicate, Aug 7, 2009 IP
  18. low.dev

    low.dev Peon

    Messages:
    3
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #18
    Could be something like this:

    <Files robots.txt>
    order allow,deny
    deny from all
    </Files>
    low.dev, Aug 7, 2009 IP
  19. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,363
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    315
    #19
    with this the robots can see but the hackers cant?
    TheSyndicate, Aug 8, 2009 IP
  20. premiumscripts

    premiumscripts Peon

    Messages:
    1,062
    Likes Received:
    46
    Best Answers:
    0
    Trophy Points:
    0
    #20
    You can also use this to your advantage;

    Put a fake "admin" folder in robots.txt and create a fake looking yoursite.com/admin -- complete with login and all but just they won't ever be able to log in (It is a fake..) -- Or you can log their details and ban them from the site, etc, whatever you want.. :)
    premiumscripts, Aug 8, 2009 IP