Anyone seen this spider?

Discussion in 'All Other Search Engines' started by NewComputer, Jul 7, 2004.

0
  1. #1
    I had a spider visit today called "Larbin". I have never heard of it. Anyone know?
     
    NewComputer, Jul 7, 2004 IP
  2. l0cke

    l0cke Active Member

    Messages:
    178
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    73
    #2
    A quick google search reveals that Larbin is an open source web crawler - http://larbin.sourceforge.net/index-eng.html
     
    l0cke, Jul 7, 2004 IP
  3. NewComputer

    NewComputer Well-Known Member

    Messages:
    2,021
    Likes Received:
    68
    Best Answers:
    0
    Trophy Points:
    188
    #3
    hmmmmm, so this could have come from anyone. I will have a look at the ip.
     
    NewComputer, Jul 7, 2004 IP
  4. megri

    megri Active Member

    Messages:
    367
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    58
    #4
    What is the way to stop this spider robots.txt
     
    megri, Jul 15, 2004 IP
  5. Touchdown

    Touchdown Peon

    Messages:
    14
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Why do you care if that Spider visits or not?
     
    Touchdown, Jul 20, 2004 IP
  6. NewComputer

    NewComputer Well-Known Member

    Messages:
    2,021
    Likes Received:
    68
    Best Answers:
    0
    Trophy Points:
    188
    #6
    A little thing called bandwidth... among other reasons.
     
    NewComputer, Jul 20, 2004 IP
  7. sarahk

    sarahk iTamer Staff

    Messages:
    28,807
    Likes Received:
    4,534
    Best Answers:
    123
    Trophy Points:
    665
    #7
    I wouldn't rely on robots.txt for anything other than legitimate search engine robots where you want to control the results shown in the search engine results.

    My site has some info on Larbin too.

    You may be better to block the bot name using your .htaccess. This post might help: http://www.mod-rewrite.com/forum/showthread.php?p=48

    Sarah
     
    sarahk, Jul 20, 2004 IP
  8. schlottke

    schlottke Peon

    Messages:
    2,185
    Likes Received:
    63
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Thanks for the link sarah, adding to my favs.
     
    schlottke, Jul 20, 2004 IP
  9. hulkster

    hulkster Peon

    Messages:
    1,705
    Likes Received:
    93
    Best Answers:
    0
    Trophy Points:
    0
    #9
    FYI FWIW: "Larbin" spidered 81 pages on the www.komar.org website last week with the same IP address of 202.9.158.10 - an example apache log entry is shown below. A reverse lookup of this IP address times out for me, but a traceroute seems to indicate it was from Singapore.

    alek

    202.9.158.10 - - [15/Jul/2004:07:46:27 -0600] "GET / HTTP/1.0" 200 7510 "-" "larbin_2.6.3 snishant@ipolicynet.com"
     
    hulkster, Jul 21, 2004 IP