1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

PHP Counter - How to not count search engine bots/spiders

Discussion in 'PHP' started by peppy, Jun 16, 2012.

  1. #1
    Greetings,

    I have a simple PHP-based counter which counts a unique visitor once per day, per page. The MySQL database records each visitors IP address so a visitor is only counted once per day.

    The problem is I think this counter is counting search engine bots and is recording a lot of false hits. Does anyone know how to prevent this?
    SEMrush
    I was thinking of trying to match the IP to a list of search engine bot IPs to exclude those visitors, but I'm guessing there's thousands of IPs to track down.

    Please help.

    Thanks
     
    Solved! View solution.
    peppy, Jun 16, 2012 IP
    SEMrush
  2. ssmm987

    ssmm987 Member

    Messages:
    180
    Likes Received:
    4
    Best Answers:
    3
    Trophy Points:
    43
    #2
    Add this to your robots.txt file:

    
    Disallow: /path/to/counter.php
    
    Code (markup):
    It's not guaranteed that it will block every bot, but it will block all respectable search engines like google, yahoo, bing etc.
     
    ssmm987, Jun 17, 2012 IP
  3. peppy

    peppy Active Member

    Messages:
    389
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    95
    #3
    I don't think this will work since the counter is a script that I built into my pages. It gets loaded along with the rest of the page in PHP.

    Let me know what you think about this.

    Thanks
    Kind regards
     
    peppy, Jun 17, 2012 IP
  4. NetStar

    NetStar Notable Member

    Messages:
    2,431
    Likes Received:
    525
    Best Answers:
    21
    Trophy Points:
    245
    #4
    You could go off the HTTP_USER_AGENT and compare it to a list of bot user agents. It'll never be 100% accurate but will prevent counting the major bots.
     
    NetStar, Jun 17, 2012 IP
  5. peppy

    peppy Active Member

    Messages:
    389
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    95
    #5
    Is there a standard, current list of the major bot IP addresses?

    Thanks
     
    peppy, Jun 17, 2012 IP
  6. #6
    I wouldn't use Bot IP Addresses. Mainly because they do in fact change and every day your list will become outdated. The maintenance would outweigh the benefit. Since you are just trying to avoid counting KNOWN bots as HITS I would simply get a list of known bot HTTP_USER_AGENTS and create a compare function to detect if the visitor is a bot.

    Like I said it will never be 100% accurate on a global scale...but for your site it may be 99.99% accurate.
     
    NetStar, Jun 17, 2012 IP
    peppy likes this.
  7. peppy

    peppy Active Member

    Messages:
    389
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    95
    #7
    Is there a standard list of known bot HTTP_USER_AGENTS? I think this could be a good plan.

    Thanks
     
    peppy, Jun 17, 2012 IP
  8. NetStar

    NetStar Notable Member

    Messages:
    2,431
    Likes Received:
    525
    Best Answers:
    21
    Trophy Points:
    245
    #8
    There are tons if you go to google and type in: HTTP_USER_AGENT Bots
     
    NetStar, Jun 17, 2012 IP
  9. peppy

    peppy Active Member

    Messages:
    389
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    95
    #9
    Thanks for the tip on trying to match user agents vs IP addresses. I'll see how well this will work.

    Kind regards
     
    peppy, Jun 19, 2012 IP
  10. stephan2307

    stephan2307 Well-Known Member

    Messages:
    1,261
    Likes Received:
    29
    Best Answers:
    7
    Trophy Points:
    150
    #10
    Alternatively you could load your counter.php file via javascript. nowadays most people have js enabled but bots don't do js.
     
    stephan2307, Jun 20, 2012 IP
  11. NetStar

    NetStar Notable Member

    Messages:
    2,431
    Likes Received:
    525
    Best Answers:
    21
    Trophy Points:
    245
    #11
    I think many of you are missing the point. He doesn't have a COUNTER script...it's not 1996.
     
    NetStar, Jun 21, 2012 IP
  12. txf

    txf Well-Known Member

    Messages:
    516
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    130
    #12
    :) LOL
     
    txf, Sep 27, 2018 IP