robot.txt to allow everything

Discussion in 'robots.txt' started by joerudy1988, May 3, 2012.

0
  1. #1
    What should the syntax be?

    Thanks.
     
    joerudy1988, May 3, 2012 IP
  2. TrinityZey

    TrinityZey Well-Known Member

    Messages:
    258
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    103
    #2
    I might not be sure, but if there isn't a robot.txt at all, wouldn't it allow everything anything? Or are you trying to force the robots to crawl through?
     
    TrinityZey, May 17, 2012 IP
  3. bsdstr

    bsdstr Peon

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    User-agent: *Disallow: / Allow: / Allow: /$
     
    bsdstr, May 17, 2012 IP
  4. joerudy1988

    joerudy1988 Peon

    Messages:
    196
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Is it better to have one to force the robots to crawl?

     
    joerudy1988, May 17, 2012 IP