1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt help

Discussion in 'HTML & Website Design' started by NewComputer, Sep 26, 2004.

0
  1. #1
    Is this a good robots.txt?

    #User-agent: lycra
    #Disallow: /

    #User-agent: *
    #Disallow: /tmp
    #Disallow: /logs

    User-agent: *
    Disallow:

    What does the
    User-agent: *
    Disallow:
    tell the spiders? That all are allowed or none?

    If you haven't figured it out by now, I am working with another company to help them. Just going over their stuff. I always thought the above should read

    User-agent: *
    Disallow: /

    To allow all others....
     
    NewComputer, Sep 26, 2004 IP
  2. vagrant

    vagrant Peon

    Messages:
    2,284
    Likes Received:
    181
    Best Answers:
    0
    Trophy Points:
    0
    #2
    User-agent: *
    Disallow: /

    = no access to all robots

    User-agent: *
    Disallow:

    gives access to all robots

    see ww w.robotstxt.org/wc/norobots.html for details
     
    vagrant, Sep 26, 2004 IP