1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Is it good to block them?

Discussion in 'All Other Search Engines' started by Nokia999, Feb 13, 2006.

0
  1. #1
    Unknown robot (identified by 'crawl')
    Unknown robot (identified by hit on 'robots.txt')
    Unknown robot (identified by 'spider')
    Unknown robot (identified by 'robot')

    Why they are unknown :confused:
    I am sure these unknown will never send any traffic :rolleyes:
    So how to block them ?
     
    Nokia999, Feb 13, 2006 IP
  2. just-4-teens

    just-4-teens Peon

    Messages:
    3,967
    Likes Received:
    168
    Best Answers:
    0
    Trophy Points:
    0
    #2
    not always good, some of them are new search bots that your stats program does not yet know about, or sometimes it is software (if you use software like webceo that will cause some of them unknown)
     
    just-4-teens, Feb 13, 2006 IP
  3. wkd

    wkd Peon

    Messages:
    64
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    You really need to monitor your web stats package. Most cPanel hosts have AWStats. You can use that or whatever stats package you have and monitor how many hits and total bandwidth is being used by bots or scripts that visit your site. One particular search engine, Gigablast, was taking something like 4 gigabytes of bandwidth! So I definitely had to block their IP address. Later I went ahead and used robots.txt file to deny them, but the problem with the robots.txt file is that there is nothing that makes any script or bot obey it. That is why you have to block their ip sometimes.
     
    wkd, Feb 13, 2006 IP