Have you heard about SBIder before?

Discussion in 'Site & Server Administration' started by tonyinabox, Apr 10, 2006.

  1. #1
    SBIder hit my robots.txt and the UA is SBIder/0.8-dev

    Should I let it crawing my website or should I ban?:confused:
     
    tonyinabox, Apr 10, 2006 IP
  2. tonyinabox

    tonyinabox Peon

    Messages:
    1,988
    Likes Received:
    42
    Best Answers:
    0
    Trophy Points:
    0
    #2
    also this one

    NutchCVS/0.8-dev
     
    tonyinabox, Apr 10, 2006 IP