unknown bots.

Discussion in 'robots.txt' started by aj845, Jul 6, 2007.

0
  1. #1
    I have these bots in my website.

    Unknown robot (identified by 'robot')
    Unknown robot (identified by 'crawl')
    Unknown robot (identified by 'spider')
    Unknown robot (identified by 'bot/' or 'bot-')

    Unknown robot (identified by 'bot/' or 'bot-')
    Unknown robot (identified by hit on 'robots.txt')
    Unknown robot (identified by 'crawl')

    Unknown robot (identified by 'robot')
    Unknown robot (identified by 'crawl')
    Unknown robot (identified by 'bot/' or 'bot-')

    Are bots good or bad. If bad how can I block them so I don't get banned from adsense.
    cheers
     
    aj845, Jul 6, 2007 IP
  2. masroor

    masroor Banned

    Messages:
    175
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #2
    if you tell ip range it will helpfull to know which bots are
     
    masroor, Jul 8, 2007 IP
  3. Dan Schulz

    Dan Schulz Peon

    Messages:
    6,032
    Likes Received:
    437
    Best Answers:
    0
    Trophy Points:
    0
    #3
    It depends. Some "bots" are search engine spiders, such as the ones used by Yahoo, Google and MSN. Other "bots" are harvesters, little programs that grab email addresses so they can be used to send spam. Yours is probably the latter case (any self-respecting search engine spider would identify itself and respect the robots.txt file), and if it is, I'd block the IP range (the whole range, not the address) that the bot is using.
     
    Dan Schulz, Jul 8, 2007 IP
  4. aj845

    aj845 Well-Known Member

    Messages:
    736
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    110
    #4
    how can i block them?
     
    aj845, Jul 10, 2007 IP
  5. Dan Schulz

    Dan Schulz Peon

    Messages:
    6,032
    Likes Received:
    437
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Does your hosting control panel have a "page" you can go to which will allow you to block them (if your control panel is cPanel you should have this)? If so, there should be instructions right there for its use.
     
    Dan Schulz, Jul 10, 2007 IP
  6. trichnosis

    trichnosis Prominent Member

    Messages:
    13,785
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #6
    if you know the ip of them , you can block all of them from your cpanel
     
    trichnosis, Aug 7, 2007 IP
  7. belgin fish

    belgin fish Well-Known Member

    Messages:
    1,544
    Likes Received:
    72
    Best Answers:
    0
    Trophy Points:
    185
    Digital Goods:
    2
    #7
    aw stats isnt showing the ip for me, how can i get their ip?
     
    belgin fish, Jan 14, 2009 IP
  8. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #8
    Get the IP adresses of these bots from your website admin panel and block those IP adresses(bots) using htaccess file..:)
     
    manish.chauhan, Jan 20, 2009 IP
  9. blue_angel

    blue_angel Well-Known Member

    Messages:
    1,174
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    130
    #9
    most of them use dynamic IP is impossible to ban them
     
    blue_angel, Jun 8, 2009 IP
  10. linkdealer

    linkdealer Active Member

    Messages:
    138
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    90
    #10
    You can block the range of IP using the htaccess
     
    linkdealer, Jun 8, 2009 IP
  11. Kezi

    Kezi Active Member

    Messages:
    68
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    91
    #11
    Old thread, but how exactly do you block the IP's for these unknown bots, if the IP address's are not showing???
     
    Kezi, Jan 17, 2011 IP
  12. Kezi

    Kezi Active Member

    Messages:
    68
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    91
    #12
    I did type it correctly above, but for some reason, it's not coming out right?? Anyway, I want to stop the unknown bots, but no IP address's are showing anywhere, so, how do you do it???
     
    Kezi, Jan 17, 2011 IP
  13. cquinndesign

    cquinndesign Peon

    Messages:
    27
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    I'm in the same boat, how do you stop these when there is no IP address. I too am using AW Stats. All it says is : Unknown robot (identified by 'bot*')
     
    cquinndesign, Jul 3, 2011 IP