Sitemaps & Robots.txt file problem

Discussion in 'Google Sitemaps' started by sas410, May 1, 2006.

  1. #1
    I've just realised, by logging into Google Sitemaps and looking in to "URLs restricted by robots.txt" - that some of my website pages are marked by Google Sitemaps at "URL restricted by robots.txt".

    Now how can this be if my robots txt file looks like so:

    # All robots will spider the domain
    User-agent: *
    Disallow:

    # Disallow directory /phpads/
    User-agent: *
    Disallow: /phpads/​

    I specifically do not block anything - howcome Google sitemaps is not reading this?!?!
     
    sas410, May 1, 2006 IP
  2. sas410

    sas410 Active Member

    Messages:
    128
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    53
    #2
    BIG oops - I just realise I posted this in the wrong forums - can some mod please move this to the Google Sitemaps forum? Many thanks - and sorry!

    I'm lacking caffeine today! :-//
     
    sas410, May 1, 2006 IP
  3. Jean-Luc

    Jean-Luc Peon

    Messages:
    601
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Hi,

    Your robots.txt file is not correct. You cannot have a set of lines saying "I do not disallow anything" and then a set of lines saying "I disallow /phpads/".

    I would remove the first set of rules.

    Jean-Luc
     
    Jean-Luc, May 2, 2006 IP
  4. sas410

    sas410 Active Member

    Messages:
    128
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    53
    #4
    You're right - that the robots.txt file contradicting itself.
    I'll fix it right away! Thanks Jean-Luc
     
    sas410, May 2, 2006 IP