robots.txt error when allowing all

Discussion in 'robots.txt' started by godsgift, Jan 9, 2012.

  1. #1
    I have a robots.txt file that is supposed to allow all bots and have no disallows. When I load up the address, it displays:

    User-agent: *Allow: /In Google Webmaster Tools, it says Line 0: http://www.xyz.com/robots.txt robots.txt file does not appear to be valid

    Then it says the text of xyz.com is:

    ?User-agent: *Allow: /

    Why does it have this question mark and why isn't it accepting this robots.txt? Does it think there is a space there?
     
    godsgift, Jan 9, 2012 IP
  2. ilook

    ilook Well-Known Member

    Messages:
    1,602
    Likes Received:
    15
    Best Answers:
    1
    Trophy Points:
    165
    #2
    The correct format in this particular case is:

    User-agent: *
    Disallow:
    Code (markup):
    This allows all bots to index your complete site.
     
    ilook, Jan 9, 2012 IP
  3. Sgarena

    Sgarena Greenhorn

    Messages:
    36
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #3
    there is no "Allow" command in robots.txt.
    if you whant to "allow all" you should remove the robots.txt or use the next code
    
    User-agent: *
    
    Code (markup):
     
    Sgarena, Jan 10, 2012 IP
  4. pankajrocks

    pankajrocks Peon

    Messages:
    32
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    yes this is right code for robots.txt file.
     
    pankajrocks, Jan 12, 2012 IP
  5. godsgift

    godsgift Peon

    Messages:
    34
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Can this potentially cause search engines to incorrectly crawl my site? Or will they even ignore the site altogether?
     
    godsgift, Jan 15, 2012 IP
  6. warnor

    warnor Active Member

    Messages:
    288
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    70
    #6
    This is perfect code for the robot txt.
     
    warnor, Jan 21, 2012 IP
  7. tiffanywilliams12i2

    tiffanywilliams12i2 Peon

    Messages:
    164
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    that config wont it works perfectly
     
    tiffanywilliams12i2, Apr 10, 2012 IP