CSF blocking Google bots

Discussion in 'Security' started by robinton, Jun 20, 2013.

0
  1. #1
    my google webmaster showing an error that
    Google couldn't access your site because of a server connectivity issue.
    i have vps dedicated server with 5 website in it.
    all website is ok but botttom error showing for just one websote
    server connectivity is ok .i think csf block google bot ..how can i solve this?
    i tried to add this code to mod_security
    # GoogleBot by user-agent…
SecRule HTTP_USER_AGENT “Google” nolog,allow
SecRule HTTP_USER_AGENT “Googlebot” nolog,allow
SecRule HTTP_USER_AGENT “GoogleBot” nolog,allow
SecRule HTTP_USER_AGENT “googlebot” nolog,allow
SecRule HTTP_USER_AGENT “Googlebot-Image” nolog,allow
##
SecRule HTTP_USER_AGENT “AdsBot-Google” nolog,allow
SecRule HTTP_USER_AGENT “Googlebot-Image/1.0″ nolog,allow
SecRule HTTP_USER_AGENT “Googlebot/2.1″ nolog,allow
SecRule HTTP_USER_AGENT “Googlebot/Test” nolog,allow
SecRule HTTP_USER_AGENT “Mediapartners-Google/2.1″ nolog,allow
SecRule HTTP_USER_AGENT “Mediapartners-Google*” nolog,allow
SecRule HTTP_USER_AGENT “msnbot” nolog,allow
    Code (markup):
    but nothing happen
     
    Solved! View solution.
    robinton, Jun 20, 2013 IP
  2. palsman

    palsman Active Member

    Messages:
    156
    Likes Received:
    1
    Best Answers:
    1
    Trophy Points:
    95
    #2
    Try to put googlebot in the file csf.rignore
    the location is : /etc/csf/csf.rignore
    This will solve the problem.
    also please check what CSF blocked and remove google IPs from the following location:
    /etc/csf/csf.deny
    /etc/csf/csf.tempip
    /etc/csf/csf.tempban

    Google IP's:
    • 216.239.32.0/19 # Googlebot
    • 64.233.160.0/19 # Googlebot
    • 72.14.192.0/18 # Googlebot
    • 209.85.128.0/17 # Googlebot
    • 66.102.0.0/20 # Googlebot
    • 74.125.0.0/16 # Googlebot
    • 66.249.64.0/19 #Googlebot
     
    palsman, Jun 20, 2013 IP
  3. robinton

    robinton Member

    Messages:
    44
    Likes Received:
    3
    Best Answers:
    1
    Trophy Points:
    33
    #3
    can i edit csf.rignore by whm?
     
    robinton, Jun 20, 2013 IP
  4. #4
    yes you can go to WHM--> Plugin--> config security server and firewall--> Quick Ignore
    you add any thing inside it, for other you can find it in the same location.
     
    palsman, Jun 20, 2013 IP
  5. jeff-ay

    jeff-ay Active Member

    Messages:
    29
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    56
    #5
    Or, you can install file explorer in WHM to browse and edit your files
    www.configserver.com/cp/cse.html‎
     
    jeff-ay, Jun 29, 2013 IP
  6. infinitnet

    infinitnet Member

    Messages:
    56
    Likes Received:
    7
    Best Answers:
    1
    Trophy Points:
    35
    #6
    I think you're using cPanel's bad default mod_security rules. CSF only blocks the bots, because it triggers your mod_security rules. One option would be to whitelist the IP ranges in CSF, another option would be to whitelist them in your mod_security rules, or fix the rules that cause the issues. However, the best solution would be to ditch the bad default rules and use a proper ruleset, such the free one from AtomiCorp's ASL rulesets.
     
    infinitnet, Jul 4, 2013 IP