Social Buttons
  1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Sitemaps & Robots.txt file problem

Discussion in 'Google Sitemaps' started by sas410, May 1, 2006.

0
  1. #1
    I've just realised, by logging into Google Sitemaps and looking in to "URLs restricted by robots.txt" - that some of my website pages are marked by Google Sitemaps at "URL restricted by robots.txt".

    Now how can this be if my robots txt file looks like so:
    SEMrush
    # All robots will spider the domain
    User-agent: *
    Disallow:

    # Disallow directory /phpads/
    User-agent: *
    Disallow: /phpads/​

    I specifically do not block anything - howcome Google sitemaps is not reading this?!?!
     
    sas410, May 1, 2006 IP
    SEMrush
  2. sas410

    sas410 Active Member

    Messages:
    128
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    53
    #2
    BIG oops - I just realise I posted this in the wrong forums - can some mod please move this to the Google Sitemaps forum? Many thanks - and sorry!

    I'm lacking caffeine today! :-//
     
    sas410, May 1, 2006 IP
  3. Jean-Luc

    Jean-Luc Peon

    Messages:
    601
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Hi,

    Your robots.txt file is not correct. You cannot have a set of lines saying "I do not disallow anything" and then a set of lines saying "I disallow /phpads/".

    I would remove the first set of rules.

    Jean-Luc
     
    Jean-Luc, May 2, 2006 IP
  4. sas410

    sas410 Active Member

    Messages:
    128
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    53
    #4
    You're right - that the robots.txt file contradicting itself.
    I'll fix it right away! Thanks Jean-Luc
     
    sas410, May 2, 2006 IP