1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Is robots.txt really safe?

Discussion in 'robots.txt' started by kevoh99, Jul 17, 2010.

0
  1. #1
    Since robots.txt resides in the root directory of a website, all you have to do is to download it by typing in the url to it.

    Isn't that a security threat?
     
    kevoh99, Jul 17, 2010 IP
  2. nimonogi

    nimonogi Active Member

    Messages:
    398
    Likes Received:
    3
    Best Answers:
    1
    Trophy Points:
    80
    #2
    No it isn't :)
     
    nimonogi, Jul 17, 2010 IP
  3. Xenlightenment-Teisho

    Xenlightenment-Teisho Peon

    Messages:
    16
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    In order to work, robots.txt needs to be world-readable. If you can't download it from http://root.tld/robots.txt, then a robot can't either.

    If you have something to hide, your robots.txt file is not the best way to conceal it. For example, whitehouse.gov/robots.txt was once famous for its size and some of its curious entries. If you see robots.txt as a "security threat", you're probably using it wrong. If you don't want people and robots to see certain pages, you should protect the pages in your .htaccess file. The robots.txt file is not a security system.
     
    Xenlightenment-Teisho, Jul 17, 2010 IP
  4. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #4
    It all depends what you are hiding through robots.txt. If you are hiding some confidential information through then it is a security threat? so better use htaccess or password protected pages to hide confidential information on the website.
     
    manish.chauhan, Jul 18, 2010 IP
  5. harry009

    harry009 Active Member

    Messages:
    93
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    73
    #5
    using robots.txt, you can provide security to your pages for crawlers then after bots will not crawle your some confidential pages.
     
    harry009, Jul 18, 2010 IP
  6. garbin

    garbin Peon

    Messages:
    128
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    The robots.txt file is not a security system. But i don't know how to do,
    Xenlightenment-Teisho says that "you should protect the pages in your .htaccess file"
    detail????
     
    garbin, Jul 19, 2010 IP
  7. Imozeb

    Imozeb Peon

    Messages:
    666
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Someone could see the robots.txt but they cant upload a new one unless they have hacked your system which by then its to late. robots.txt only blocks SE spiders from seeing the page. You can still navigate to it by typing in the url.
     
    Imozeb, Jul 22, 2010 IP
  8. cahbagoes

    cahbagoes Peon

    Messages:
    27
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    wow, what is .htaccess file?? fr nubie blogger
     
    cahbagoes, Jul 22, 2010 IP
  9. masini

    masini Peon

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Yes, but how can I change the robots.txt file. Since FTP is not. Can someone give me some details, too? I want something out there to the public but do not know how to edit this file
    ___________________________________________________________
    piese auto online auto dealers
     
    masini, Jul 23, 2010 IP
  10. Mikesloug

    Mikesloug Guest

    Messages:
    150
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #10
    I like your post.Is this file is important for my site or not in SEo?
     
    Mikesloug, Aug 2, 2010 IP
  11. josephs

    josephs Peon

    Messages:
    15
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Thanks for the interesting post.
     
    josephs, Aug 3, 2010 IP
  12. SteveDakine

    SteveDakine Peon

    Messages:
    14
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    I agree with manish.chauhan here, robots.txt is NOT the the absolute best method for stopping bots. To be sure, use htaccess or password protected pages. Also, at a lower effectiveness level, nofollow is now not paid attention to by search engines like Google. The search engines decided to be the ones who decide what they crawl, not the webmasters. PageRank sculpting is fairly redundant now. Just my 2 pence!
     
    SteveDakine, Aug 22, 2010 IP
  13. seo.lover

    seo.lover Peon

    Messages:
    132
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    it is a file containing redirection code basically
     
    seo.lover, Aug 29, 2010 IP