Is it safe to block Java User-Agents?

Discussion in 'Traffic Analysis' started by Krnl, Mar 1, 2006.

0
  1. #1
    I get several hits per day from Java user-agents that seem to be doing nothing useful. My question is: Will blocking all Java user-agents from my site affect the crawling behavior of "friendly bots" like googlebot, msn, yahoo, etc?

    Here is an excerpt from my access_log indicating the type of hits that I'm talking about:

    80.57.195.103 - - [01/Mar/2006:13:20:22 -0500] "GET /cookie_usage.php HTTP/1.1" 200 38156 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:20:22 -0500] "GET /includes/general.js HTTP/1.1" 200 4507 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:20:22 -0500] "GET //smarticon.geotrust.com/si.js HTTP/1.1" 404 235 "-" "Java/1.5.0_04"

    -- Why does this thing prefix https requests with a "/"? Is this a known vulnerability on some webservers?

    80.57.195.103 - - [01/Mar/2006:13:20:23 -0500] "GET /https://all-in-general.com/account.php HTTP/1.1" 404 244 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:20:23 -0500] "GET /https://all-in-general.com/checkout_shipping.php HTTP/1.1" 404 254 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:20:23 -0500] "GET /https://all-in-general.com/create_account.php HTTP/1.1" 404 251 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:20:23 -0500] "GET /https://all-in-general.com/login.php HTTP/1.1" 404 242 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:20:24 -0500] "GET /https://all-in-general.com/account_notifications.php HTTP/1.1" 404 258 "-" "Java/1.5.0_04"

    -- Every time this user agent hits my site (from multiple IP addresses ranging from .EU to .US based sites), it requests the following two pages:

    80.57.195.103 - - [01/Mar/2006:13:20:24 -0500] "GET /https://www.secure-server-hosting.com/secutran/secureforms/sh208794/mortgage.htm HTTP/1.1" 404 286 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:20:24 -0500] "GET /https://www.secure-server-hosting.com/secutran/secureforms/sh208794/homeowner-loan.htm HTTP/1.1" 404 292 "-" "Java/1.5.0_04"

    I think I should block this thing, but want to ensure that it will not affect my SEO negatively by doing so.

    80.57.195.103 - - [01/Mar/2006:13:20:31 -0500] "GET /index.php?cPath=1 HTTP/1.1" 200 49111 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:20:39 -0500] "GET /product_info.php?products_id=19695 HTTP/1.1" 200 44757 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:20:46 -0500] "GET /specials.php?page=2 HTTP/1.1" 200 43704 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:20:47 -0500] "GET /product_reviews.php?products_id=10999&action=buy_now HTTP/1.1" 302 5 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:20:53 -0500] "GET /cookie_usage.php HTTP/1.1" 200 38227 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:20:59 -0500] "GET /index.php?cPath=108 HTTP/1.1" 200 45292 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:21:07 -0500] "GET /product_info.php?products_id=2047 HTTP/1.1" 200 44874 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:21:14 -0500] "GET /specials.php?page=3 HTTP/1.1" 200 38751 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:21:15 -0500] "GET /product_reviews.php?products_id=10999&action=notify HTTP/1.1" 302 5 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:21:22 -0500] "GET /cookie_usage.php HTTP/1.1" 200 38145 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:21:29 -0500] "GET /index.php?cPath=113 HTTP/1.1" 200 40946 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:21:38 -0500] "GET /product_info.php?products_id=2391 HTTP/1.1" 200 44817 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:21:44 -0500] "GET /product_reviews.php?products_id=19695 HTTP/1.1" 200 38949 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:21:44 -0500] "GET /product_reviews_write.php?products_id=19695 HTTP/1.1" 302 5 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:21:50 -0500] "GET /specials.php?page=1 HTTP/1.1" 200 43706 "-" "Java/1.5.0_04"
    80.57.195.103 - - [01/Mar/2006:13:21:58 -0500] "GET /index.php?cPath=118 HTTP/1.1" 200 47590 "-" "Java/1.5.0_04"

    ANy information relating to the logs above will be much appreciated.

    Thanks for your time.

    KrnlPanic
     
    Krnl, Mar 1, 2006 IP