1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Is robot.txt a Hacking tool?

Discussion in 'Security' started by TheSyndicate, Jul 16, 2009.

  1. #1
    Webmasters wanting to exclude search engine robots from certain parts of their site often choose the use of a robot.txt file on the root of the server. This file basicly tells the bot which directories are supposed to be off-limits. An attacker can easily obtain that information by very simply opening that plain text file in his browser. Webmasters should *never* rely on this for real security issues. Google helps the attacker by allowing a search for the "disallow" keyword.

    The Google Hacking Database (GHDB) appears courtesy of the Google Hacking community.

    what do you think?
     
    TheSyndicate, Jul 16, 2009 IP
  2. Dixie

    Dixie Guest

    Messages:
    46
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #2
    It's not a Hacking tool per se, it can just make a hack a lot easier if you have a file telling you where you're not supposed to go. ;)
     
    Dixie, Jul 16, 2009 IP
  3. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #3
    so should you use it or not
     
    TheSyndicate, Jul 16, 2009 IP
  4. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #4
    If you are relying on robots.txt file to keep people out of places they shouldn't, you have much bigger problems.
     
    digitalpoint, Jul 16, 2009 IP
  5. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #5
    It is nice to know DP have good music taste. No i do not trust robot text to keep me out i just saw it when i scan my files for security. So i guess i can keep it the hackers will find the files anyway
     
    TheSyndicate, Jul 17, 2009 IP
  6. justdoit1

    justdoit1 Peon

    Messages:
    100
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #6
    An important thing is that you should never place sensitive data inside the web root directory.
     
    justdoit1, Jul 17, 2009 IP
  7. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #7
    if you have a cpanel and many domains in the same root stuff happen to come there but that has noting to do with robot.txt
     
    TheSyndicate, Jul 17, 2009 IP
  8. kunnusingh

    kunnusingh Well-Known Member

    Messages:
    648
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    145
    #8
    Use permission. Set to ony excute so people cant read your robots.txt
     
    kunnusingh, Jul 20, 2009 IP
  9. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #9
    Use permission. Set to ony excute so people cant read your robots.txt

    what do you mean with this
     
    TheSyndicate, Jul 20, 2009 IP
  10. Dixie

    Dixie Guest

    Messages:
    46
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #10
    ....Does that not make robots.txt pointless?
     
    Dixie, Jul 20, 2009 IP
  11. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #11
    and he got banned ... o well not it does not the good once still follow the rules. Google follow the robot.txt i guess so save loads of bandwith not to allow them to some places.
     
    TheSyndicate, Jul 20, 2009 IP
  12. anuradhan

    anuradhan Active Member

    Messages:
    371
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    65
    #12
    no robot.txt is not a hacking tool :cool:.it is a file associated with the server, which stores the information such as meta data about an website.so that when a query is send to the server,it checks the robot.txt and produce the result with in a small period of time.
     
    anuradhan, Jul 20, 2009 IP
  13. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #13
    well the hackers can see, well maybe they can see anyway what ever .-.
     
    TheSyndicate, Aug 7, 2009 IP
  14. Qryztufre

    Qryztufre Prominent Member

    Messages:
    6,071
    Likes Received:
    491
    Best Answers:
    0
    Trophy Points:
    300
    #14
    I think I agree with Digitalpoint...

    Hackers can and likely do use Robot.txt to hack sites, but only sites from people that are not properly using it. A bot can not access a page/file they do not otherwise have access too, so by disallowing ANYONE but you into certain areas of your server also disallows bots. Another key is to not link to areas bots should not be getting into...
     
    Qryztufre, Aug 7, 2009 IP
  15. scylla

    scylla Notable Member

    Messages:
    1,025
    Likes Received:
    33
    Best Answers:
    1
    Trophy Points:
    225
    #15
    all you need is simple .htaccess code to prevent people from even looking at robots.txt to begin with.

    If the ip addresses do not match the ones that the bot use, send them to another page.
     
    Last edited: Aug 7, 2009
    scylla, Aug 7, 2009 IP
  16. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #16
    could you give me a example what you mean?
     
    TheSyndicate, Aug 7, 2009 IP
  17. low.dev

    low.dev Peon

    Messages:
    3
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #17
    Could be something like this:

    <Files robots.txt>
    order allow,deny
    deny from all
    </Files>
     
    low.dev, Aug 7, 2009 IP
  18. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #18
    with this the robots can see but the hackers cant?
     
    TheSyndicate, Aug 8, 2009 IP
  19. premiumscripts

    premiumscripts Peon

    Messages:
    1,062
    Likes Received:
    48
    Best Answers:
    0
    Trophy Points:
    0
    #19
    You can also use this to your advantage;

    Put a fake "admin" folder in robots.txt and create a fake looking yoursite.com/admin -- complete with login and all but just they won't ever be able to log in (It is a fake..) -- Or you can log their details and ban them from the site, etc, whatever you want.. :)
     
    premiumscripts, Aug 8, 2009 IP
  20. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #20
    hahaha LOL you mean when they click log in a ban record the IP and ban them?
     
    TheSyndicate, Aug 13, 2009 IP