1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

is a blank robots file ok?

Discussion in 'robots.txt' started by fryman, Jul 13, 2005.

  1. #1
    My logs are getting filled with 404 hits looking for the robots.txt file, is it ok if I just upload a blank one?
     
    fryman, Jul 13, 2005 IP
  2. kdb003

    kdb003 Active Member

    Messages:
    150
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    73
    #2
    i dont think 404s on robots.txt files are really a problem
    i think you will be ok either way
     
    kdb003, Jul 13, 2005 IP
  3. fryman

    fryman Kiss my rep

    Messages:
    9,605
    Likes Received:
    777
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Yeah, I just don't want my log filled with this error
     
    fryman, Jul 13, 2005 IP
  4. magellan

    magellan Guest

    Messages:
    155
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Yes a blank robots.txt is fine if you don't need to block any spiders and you want to reduce the number of 404 errors in your logs.
     
    magellan, Jul 14, 2005 IP
  5. subseo

    subseo Guest

    Messages:
    652
    Likes Received:
    38
    Best Answers:
    0
    Trophy Points:
    0
    #5
    That's exactly what I did, so that to have emptier logs. An alternative is to put in the two lines or so, that "allow everything", don't have it handy, but if you search for it, it is everywhere. Anyway, with blank robots.txt, you are just fine.
     
    subseo, Jul 14, 2005 IP
  6. johnt

    johnt Peon

    Messages:
    178
    Likes Received:
    21
    Best Answers:
    0
    Trophy Points:
    0
    #6
    User-agent: *
    Disallow:

    Will tell every bot that they can read everything

    John
     
    johnt, Jul 14, 2005 IP
  7. expat

    expat Stranger from a far land

    Messages:
    873
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    0
    #7
    yes this will work if you are concerned about 404 from robots

    A nice tool is to use a customized 404 that sends you e-mail based on referrer.
    Thus you know immediately when users hit a 404 and can react and you can filter out bot's so you also know when undesirables start hammering your site - which is a good time to use a redirect based on referrer to screw them up a bit - love to send them to nonexisting (dynamic) pages with fake e-mails etc....


    Expat
     
    expat, Jul 14, 2005 IP
  8. exam

    exam Peon

    Messages:
    2,434
    Likes Received:
    120
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Yes see here
     
    exam, Jul 16, 2005 IP
  9. Enigma121

    Enigma121 Peon

    Messages:
    40
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Our investigations show that without a robots.txt file MSN will not index you!

    We added a blank file and voila a listing appeared within a matter of hours. Vitally important to have one in place from that point of view then...
     
    Enigma121, Sep 5, 2005 IP
  10. fryman

    fryman Kiss my rep

    Messages:
    9,605
    Likes Received:
    777
    Best Answers:
    0
    Trophy Points:
    0
    #10
    A robots file doesn't have anything to do with MSN not indexing you
     
    fryman, Sep 5, 2005 IP
  11. Enigma121

    Enigma121 Peon

    Messages:
    40
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Enigma121, Sep 5, 2005 IP
  12. fryman

    fryman Kiss my rep

    Messages:
    9,605
    Likes Received:
    777
    Best Answers:
    0
    Trophy Points:
    0
    #12
    I can get any site indexed at MSN in 24 hours, doesn't mater at all if it has robots file or not.

    Don't believe anything you read...
     
    fryman, Sep 5, 2005 IP
  13. explorer

    explorer Well-Known Member

    Messages:
    463
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    110
    #13
    Enigma121, I would like to add my own experience to Fryman's. I have sites at the top of MSN searches for slightly worthwhile (but not difficult) search terms. These sites have no robots.txt files.

    Fryman, I also have sites doing well with blank robots.txt files.

    I hope that helps.
     
    explorer, Sep 5, 2005 IP
  14. Enigma121

    Enigma121 Peon

    Messages:
    40
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #14
    It might have been pure co-incidence in our case, as I added the robots.txt file at around the same time I started to build backlinks. If that's the case then the timing in our case was unfortunate.
     
    Enigma121, Sep 5, 2005 IP
  15. Willy

    Willy Peon

    Messages:
    281
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    0
    #15
    Hehe, I sure hope you meant "don't believe everything you read"... ;)
     
    Willy, Sep 5, 2005 IP
  16. fryman

    fryman Kiss my rep

    Messages:
    9,605
    Likes Received:
    777
    Best Answers:
    0
    Trophy Points:
    0
    #16
    yeah, but I couldn't edit my post anymore
     
    fryman, Sep 5, 2005 IP
  17. INV

    INV Peon

    Messages:
    1,686
    Likes Received:
    101
    Best Answers:
    0
    Trophy Points:
    0
    #17
    1.Many people write articles just for content with twisted so called facts. When somebody makes a ridiculous claim that an engine won't index your website for not using robots.txt, its bogus!

    2.I would totally agree with Fryman on this one. You don't need robots.txt at all for a regular website to function well in a search engine index.
     
    INV, Sep 9, 2005 IP
  18. Shoemoney

    Shoemoney $

    Messages:
    4,475
    Likes Received:
    585
    Best Answers:
    0
    Trophy Points:
    295
    #18
    in my experience a blank robots.txt file is fine. I do not have experience with not having one.
     
    Shoemoney, Sep 9, 2005 IP
  19. minstrel

    minstrel Illustrious Member

    Messages:
    15,092
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    460
    #19
    According to the robots.txt standard, a blank robots.txt file is treated the same as no robots.txt file (except you won't see an error 404 in your logs in case that worries you).

    I've just never seen the point - in the time it takes you to create and upload an empty robots.txt file, you can create one that says:

    User-agent: *
    Disallow: 
    
    Code (markup):
    and then there's no question at all...
     
    minstrel, Oct 1, 2005 IP
  20. lorien1973

    lorien1973 Notable Member

    Messages:
    12,210
    Likes Received:
    602
    Best Answers:
    0
    Trophy Points:
    260
    #20
    I have 3 yahoo stores. I cannot upload a robots.txt file to any of them. I cannot have any file in the root domain actually (its a pain in the butt) that yahoo doesn't create.

    I am indexed by MSN just fine. Robots.txt does not factor into it at all.

    I was just browsing the log files of a new site I made. I noticed a ton of errors for the robots.txt file; and the favicon.ico file as well. I just made them and uploaded them, just to have a smaller error log.
     
    lorien1973, Oct 1, 2005 IP