1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Seen this spider before?

Discussion in 'All Other Search Engines' started by SEbasic, Aug 4, 2004.

  1. #1
    Hi there...

    Has anyone ever seen this before?

    When: Tue Aug 3 10:27:42 2004 Who: Mozilla/3.0 NAVIO (dtv0) Where: /(Details Removed)/(keyword)-(keyword)-chateau.html From: 192.220.74.15

    It's been visiting one of my sites a HELL of a lot...

    Constantly trying to visit certian pages over and over...

    Not too sure whether to ban it.

    Also, has anyone seen much activity from Slurp! Recently? It's been going CRAZY on my site.
    SEMrush
    DEEP crawls. Lots of pages have been indexed within the last 3 days, pretty much non-stop.

    Anyone else seen anything similiar recently?
     
    SEbasic, Aug 4, 2004 IP
    SEMrush
  2. anthonycea

    anthonycea Banned

    Messages:
    13,378
    Likes Received:
    342
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Can't recall that one, but there are so many now that you will need a spider identity site to figure them out, I know that I had bookmarked a few of those computer robotic sites.

    Shawn or someone else could give you a list of these robots or a website or two that will help you identify them.

    Just do a search on "robot spiders" and you will find the websites.

    I would not ban any of them without knowing who they are, a lot of universities have their own spiders, even a lot of large companies do their own crawling now.

    In your case SE, since you have so much money maybe Richard Branson is keeping an eye on you for a possible buy out :D
     
    anthonycea, Aug 4, 2004 IP
  3. SEbasic

    SEbasic Peon

    Messages:
    6,318
    Likes Received:
    318
    Best Answers:
    0
    Trophy Points:
    0
    #3
    hehe - That would be nice ;)
     
    SEbasic, Aug 4, 2004 IP