1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robot.txt to allow everything

Discussion in 'robots.txt' started by joerudy1988, May 3, 2012.

0
  1. #1
    What should the syntax be?

    Thanks.
     
    joerudy1988, May 3, 2012 IP
  2. TrinityZey

    TrinityZey Active Member

    Messages:
    258
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    53
    #2
    I might not be sure, but if there isn't a robot.txt at all, wouldn't it allow everything anything? Or are you trying to force the robots to crawl through?
     
    TrinityZey, May 17, 2012 IP
  3. bsdstr

    bsdstr Peon

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    User-agent: *Disallow: / Allow: / Allow: /$
     
    bsdstr, May 17, 2012 IP
  4. joerudy1988

    joerudy1988 Peon

    Messages:
    196
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Is it better to have one to force the robots to crawl?

     
    joerudy1988, May 17, 2012 IP