1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt error when allowing all

Discussion in 'robots.txt' started by godsgift, Jan 9, 2012.

0
  1. #1
    I have a robots.txt file that is supposed to allow all bots and have no disallows. When I load up the address, it displays:

    User-agent: *Allow: /In Google Webmaster Tools, it says Line 0: http://www.xyz.com/robots.txt robots.txt file does not appear to be valid

    Then it says the text of xyz.com is:

    ?User-agent: *Allow: /

    Why does it have this question mark and why isn't it accepting this robots.txt? Does it think there is a space there?
     
    godsgift, Jan 9, 2012 IP
  2. ilook

    ilook Well-Known Member

    Messages:
    1,602
    Likes Received:
    15
    Best Answers:
    1
    Trophy Points:
    165
    #2
    The correct format in this particular case is:

    User-agent: *
Disallow:
    Code (markup):
    This allows all bots to index your complete site.
     
    ilook, Jan 9, 2012 IP
  3. Sgarena

    Sgarena Greenhorn

    Messages:
    36
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #3
    there is no "Allow" command in robots.txt.
    if you whant to "allow all" you should remove the robots.txt or use the next code
    
User-agent: *
    Code (markup):
     
    Sgarena, Jan 10, 2012 IP
  4. pankajrocks

    pankajrocks Peon

    Messages:
    32
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    yes this is right code for robots.txt file.
     
    pankajrocks, Jan 12, 2012 IP
  5. godsgift

    godsgift Peon

    Messages:
    34
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Can this potentially cause search engines to incorrectly crawl my site? Or will they even ignore the site altogether?
     
    godsgift, Jan 15, 2012 IP
  6. warnor

    warnor Active Member

    Messages:
    288
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    70
    #6
    This is perfect code for the robot txt.
     
    warnor, Jan 21, 2012 IP
  7. tiffanywilliams12i2

    tiffanywilliams12i2 Peon

    Messages:
    164
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    that config wont it works perfectly
     
    tiffanywilliams12i2, Apr 10, 2012 IP