1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

WISEnutbot SE Bot

Discussion in 'All Other Search Engines' started by swapshop, Jul 18, 2006.

0
  1. #1
    How do we block or slow down WISEnutbot SE Bot?

    sv-crawlfw4.looksmart.com

    Its been hammering the site for 4-5 days now
     
    swapshop, Jul 18, 2006 IP
  2. hans

    hans Well-Known Member

    Messages:
    2,923
    Likes Received:
    126
    Best Answers:
    1
    Trophy Points:
    173
    #2
    1st of all
    ask yourself the question
    do you want more traffic
    or do you prefer to shut down your site or have it secret - just for friends

    since you may prefer second option you may disallow ALL bots in your robots.txt - this is done by

    User-agent: *
    Disallow: /

    if you like SOME traffic fut only from certain SE and NOT wisenut and all its sister SE's looksmart, zeal directory
    then you may exclude in your robots.txt just that one bot
    a full robots.txt tutorial is here

    http://www.clockwatchers.com/robots_main.html

    that also includes the solution to use .htaccess for blocking a particular bot - sme bots may disobey robots.tyt

    robots.txt for one bot exclusion is

    User-agent: BotName
    Disallow: /

    replace BotName with the bot you want to exclude
     
    hans, Jul 18, 2006 IP
    EveryQuery likes this.
  3. Cobalt64

    Cobalt64 Peon

    Messages:
    183
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Thanks for the information hans, might get me somewhere closer to understanding how all this bot stuff works.

    Sorry to intrude, but why would anyone want to stop or slow the indexing of their site?
     
    Cobalt64, Jul 20, 2006 IP
  4. hans

    hans Well-Known Member

    Messages:
    2,923
    Likes Received:
    126
    Best Answers:
    1
    Trophy Points:
    173
    #4
    bots are the little "dwarfs" that bring your pages to the SE who then bring you traffic and thus happiness into your life ...

    be happy to have bots visiting your site
    your server can surely handle ten thousands of bot visits a day
     
    hans, Jul 20, 2006 IP
  5. Cobalt64

    Cobalt64 Peon

    Messages:
    183
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I figured that they would only improve traffic, hence why I wondered why somebody may wish to delay or stop bots indexing their sites altogether :)

    I'm actually getting a fair few hits now from bots, and whilst I'm only really seeing the big names, I certainly won't be complaining about it!
     
    Cobalt64, Jul 20, 2006 IP
  6. 50plus

    50plus Guest

    Messages:
    234
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #6
    That depends on what bot is crawling ;)

    Not all bots crawl your site for the sole purpose of indexing your content and sending you visitors. Many bots are not search engine related and only crawl for reasons of their own, so they use your bandwidth and grab your content without ever sending you a single visitor.
     
    50plus, Jul 22, 2006 IP
  7. Absolutus

    Absolutus Peon

    Messages:
    233
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #7
    For some sites I placed it in my robots.txt

    User-agent:WISENutbot
    Disallow:/


    If someone have a 50000 pages site all generated from php or perl and mysql,
    for the few visits a small engine can bring is better to stop instead of slowing the server.
    I considered looksmart a small engine because for all my sites send very little traffic if any before disallowing it.
     
    Absolutus, Jul 22, 2006 IP
  8. swapshop

    swapshop Peon

    Messages:
    656
    Likes Received:
    34
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Ok next bot hammering us is

    Search results for: 209.85.32.65


    OrgName: Everyones Internet
    OrgID: EVRY
    Address: 390 Benmar
    Address: Suite 200
    City: Houston
    StateProv: TX
    PostalCode: 77060
    Country: US

    NetRange: 209.85.0.0 - 209.85.127.255
    CIDR: 209.85.0.0/17
    NetName: EVRY-BLK-15
    NetHandle: NET-209-85-0-0-1
    Parent: NET-209-0-0-0-0
    NetType: Direct Allocation
    NameServer: NS1.EV1.NET
    NameServer: NS2.EV1.NET
    Comment:
    RegDate: 2005-12-14
    Updated: 2006-07-17

    Are they good? I dont see why they have done 1600+ they just seem to be using b/w. The referr is XP I.E 6.0?
     
    swapshop, Jul 22, 2006 IP
  9. Absolutus

    Absolutus Peon

    Messages:
    233
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Absolutus, Jul 22, 2006 IP
  10. swapshop

    swapshop Peon

    Messages:
    656
    Likes Received:
    34
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Great Articles thanks they help to explain a great way around the bad bots.
     
    swapshop, Jul 23, 2006 IP
  11. Absolutus

    Absolutus Peon

    Messages:
    233
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #11
    My advice is to make a comprensive robots.txt for robots that follow the guidelines of robot.txt, so that only the bad robots use the trap, so you can save more cpu usage by utilizing mod_rewrite only for bad robots.
     
    Absolutus, Jul 23, 2006 IP