Things you should be doing to block bad BOTS.

Hello Friends

Been too busy with all the work, but I found something that I want to share.
Hope this can be useful for other members

When you want to know what is Robots.txt and what are the basic things need to be done to maintain this file , read this article he have covered it pretty much nicely. I will cover the couple of major issues not added in the above article...

We are talking about How many BOTS are out there and how many more are getting creating everyday. Not All Bots want to crawl your website ,t hey want to submit forms, put comments and steel your data.

I recently faced a similar trouble I have been attacked by BOTS from all over the world.

There are bad robots that can break your server, if you allow all the robots to crawl, you may find up to 50% of your resources are already used.

Here is the list of robots that I don't allow to get into my sites
http://www.business2sell.com.au/robots.txt

Sometimes even this is not enough not everyone plays by the rules, so more precautions need to be taken, in your code.

To resolve that Issue I discovered in PHP ( you can certainly look for similar situation in other coding practices ) Google and Bing does provide where they are coming from by using $_SERVER['HTTP_FROM'].

For this you need to check if the user is accessing your pages very fast and keep doing it over and over again. Lets say we check if the user is going from last pager to new page in less then 3 sec and he have done it 20 times+, I consider them bad users or BOTS.

Now you can certainly create a code something like this, People who things that code need tweaking are more than happy to put their input. I wish to learn too.

// if this is bing or google the condition would not work
if($_SERVER['HTTP_FROM']==""){
  if (!isset($_SESSION['page_count_ts_new'])){
    $_SESSION['page_count_ts_new'] = time();
    $_SESSION['bad_page_count'] = 0;
  }else{
    // recording the last timestamp from new
    $_SESSION['page_count_ts_last'] = $_SESSION['page_count_ts_new'];
    // and assigning a new timestamp
    $_SESSION['page_count_ts_new'] = time();  
  
    // time to check difference between old and new timestamps
    if(($_SESSION['page_count_ts_last'])>($_SESSION['page_count_ts_new']-3)){
        $_SESSION['bad_page_count'] = $_SESSION['bad_page_count']+1;      
    }
  
    if($_SESSION['bad_page_count']>20){
        header("Location: http://www.yourdomainame.com/ip-blocked-temp.php");  
    }  
  
  }
}else{
    // this is from google bot or Bing
}

PHP:

you can always add some more code like adding the bad ip in the temp_block_table and block them with firewall if they are keep hitting the server no matter what.

Hope this help and have great day

Manish