Does the Microsoft bot ignore the robots.txt file?

Discussion in 'Bing' started by TopDraw, Jul 13, 2007.

0
  1. #1
    Hi guys,

    I want to ask if you noticed that the MSN bot ignores the robot.txt files and just goes ahead and indexes the things you don't want to be indexed?

    Just checked and I see that some pages that I put the code in the robots.txt file as:

    User-agent: *
    Disallow: /contact/
    Disallow:/about/

    is still ignored by the MSN bot which goes on to index them anyway! :eek:

    I thought that piece of text above was to be obeyed by law abiding bots?
     
    TopDraw, Jul 13, 2007 IP
  2. Geoffrey

    Geoffrey Banned

    Messages:
    403
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #2
    No, I can confirm that it doesn't from experience the other day.
    They're not allowed to ignore it anyway, maybe you've put the wrong coding or something.
     
    Geoffrey, Jul 13, 2007 IP
  3. TopDraw

    TopDraw Active Member

    Messages:
    314
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    58
    #3
    Actually that piece of code

    is the first line in my robots.txt file and has been there from nearly the first day.

    But the search results in MSN shows those 2 pages... right at the top as well :confused:

    MSN is giving those 2 pages a lot of extra love...:rolleyes:
     
    TopDraw, Jul 13, 2007 IP
  4. DeViAnThans3

    DeViAnThans3 Peon

    Messages:
    785
    Likes Received:
    83
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I found the following possible solution on the internet.
    Normally the robots.txt you used, should work, but maybe this works better; depends on how MSN bot exactly is programmed. :)

    Somewhere else, I found the following.
     
    DeViAnThans3, Jul 18, 2007 IP