Does anyone know if Yahoo! pays attention to robots.txt when it spiders a site?

Discussion in 'Search Engine Optimization' started by NiceFairPrice, Feb 20, 2009.

  1. #1
    Does anyone know if Yahoo! pays attention to robots.txt when it spiders a site?
     
    NiceFairPrice, Feb 20, 2009 IP
  2. jitendraag

    jitendraag Notable Member

    Messages:
    3,982
    Likes Received:
    324
    Best Answers:
    1
    Trophy Points:
    270
    #2
    jitendraag, Feb 20, 2009 IP
  3. NiceFairPrice

    NiceFairPrice Peon

    Messages:
    56
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Thanks, jitendraag, for the great info.

    At least Yahoo's "Slurp" spider does not slurp up my copyrighted graphics like googlebot seems to have a voracious appitite for graphics to add to google's graphics directory. :) After working hard to create unique graphics it gets pretty annoying to have google redistribute my graphics for others to download and use them from the google graphics directory. It looks like using robots.txt to keep googlebot out of my graphics folder did the trick.
     
    NiceFairPrice, Feb 24, 2009 IP
  4. contentboss

    contentboss Peon

    Messages:
    3,241
    Likes Received:
    54
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I tested it once - protecting one folder in robots immediately stopped the slurp, so I'd say yes, they follow the standard.
     
    contentboss, Feb 25, 2009 IP
  5. portals

    portals Peon

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I made a robots file and thought it was working till i tested it from MSN site to fine out I was missing things in it.
    So it pays to check it with MSN first.
     
    portals, Feb 25, 2009 IP
  6. kutekutta

    kutekutta Peon

    Messages:
    807
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Every SE are pay attention on robot.txt.
     
    kutekutta, Feb 25, 2009 IP