1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

CSF blocking Google bots

Discussion in 'Security' started by robinton, Jun 20, 2013.

  1. #1
    my google webmaster showing an error that
    Google couldn't access your site because of a server connectivity issue.
    i have vps dedicated server with 5 website in it.
    all website is ok but botttom error showing for just one websote
    server connectivity is ok .i think csf block google bot ..how can i solve this?
    i tried to add this code to mod_security
    # GoogleBot by user-agent…
    SecRule HTTP_USER_AGENT “Google” nolog,allow
    SecRule HTTP_USER_AGENT “Googlebot” nolog,allow
    SecRule HTTP_USER_AGENT “GoogleBot” nolog,allow
    SecRule HTTP_USER_AGENT “googlebot” nolog,allow
    SecRule HTTP_USER_AGENT “Googlebot-Image” nolog,allow
    ##
    SecRule HTTP_USER_AGENT “AdsBot-Google” nolog,allow
    SecRule HTTP_USER_AGENT “Googlebot-Image/1.0″ nolog,allow
    SecRule HTTP_USER_AGENT “Googlebot/2.1″ nolog,allow
    SecRule HTTP_USER_AGENT “Googlebot/Test” nolog,allow
    SecRule HTTP_USER_AGENT “Mediapartners-Google/2.1″ nolog,allow
    SecRule HTTP_USER_AGENT “Mediapartners-Google*” nolog,allow
    SecRule HTTP_USER_AGENT “msnbot” nolog,allow 
    Code (markup):
    but nothing happen
    SEMrush
     
    Solved! View solution.
    robinton, Jun 20, 2013 IP
    SEMrush
  2. palsman

    palsman Active Member

    Messages:
    156
    Likes Received:
    1
    Best Answers:
    1
    Trophy Points:
    95
    #2
    Try to put googlebot in the file csf.rignore
    the location is : /etc/csf/csf.rignore
    This will solve the problem.
    also please check what CSF blocked and remove google IPs from the following location:
    /etc/csf/csf.deny
    /etc/csf/csf.tempip
    /etc/csf/csf.tempban

    Google IP's:
    • 216.239.32.0/19 # Googlebot
    • 64.233.160.0/19 # Googlebot
    • 72.14.192.0/18 # Googlebot
    • 209.85.128.0/17 # Googlebot
    • 66.102.0.0/20 # Googlebot
    • 74.125.0.0/16 # Googlebot
    • 66.249.64.0/19 #Googlebot
     
    palsman, Jun 20, 2013 IP
  3. robinton

    robinton Member

    Messages:
    44
    Likes Received:
    3
    Best Answers:
    1
    Trophy Points:
    33
    #3
    can i edit csf.rignore by whm?
     
    robinton, Jun 20, 2013 IP
  4. #4
    yes you can go to WHM--> Plugin--> config security server and firewall--> Quick Ignore
    you add any thing inside it, for other you can find it in the same location.
     
    palsman, Jun 20, 2013 IP
  5. jeff-ay

    jeff-ay Active Member

    Messages:
    29
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    56
    #5
    Or, you can install file explorer in WHM to browse and edit your files
    www.configserver.com/cp/cse.html‎
     
    jeff-ay, Jun 29, 2013 IP
  6. infinitnet

    infinitnet Member

    Messages:
    56
    Likes Received:
    7
    Best Answers:
    1
    Trophy Points:
    35
    #6
    I think you're using cPanel's bad default mod_security rules. CSF only blocks the bots, because it triggers your mod_security rules. One option would be to whitelist the IP ranges in CSF, another option would be to whitelist them in your mod_security rules, or fix the rules that cause the issues. However, the best solution would be to ditch the bad default rules and use a proper ruleset, such the free one from AtomiCorp's ASL rulesets.
     
    infinitnet, Jul 4, 2013 IP