Search Engine Bots User Agent

Discussion in 'Programming' started by rederick, Dec 29, 2005.

  1. #1
    Hi, I have a page that requries the user to be "registered" to follow the links on this page. Basically they have to have a cookie set. What I want to do is detect the search engines user agent and let the search engines follow the links without haveing the cookies set. How would I find out what the user agent of the googlebot, and the others.

    something like this

    if ($_SERVER['HTTP_USER_AGENT'] == "googlebot"){
    let them in
    }

    Thanks

    Red.
     
    rederick, Dec 29, 2005 IP
  2. Nitin M

    Nitin M White/Gray/Black Hat

    Messages:
    640
    Likes Received:
    93
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Yeah, you could do something like that but depending on how secure you want the system to be, it may not be a great solution. There are easy ways for people to change their user agent.

    If you need it to be more secure, then going off IP is the way to go but since their is no published list of bot ip address it needs constant updating. Also, you'll want to make sure you use the nocache tag on the page otherwise anyone can just check google's cache and see your "hidden" content.
     
    Nitin M, Dec 29, 2005 IP
  3. rederick

    rederick Peon

    Messages:
    128
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #3
    To explain myself a little better.... The people who'll be viewing this website will not be able to spoof their user agent, and if they do they can it's not sensitive data or anything, we're just hoping to get their email address :)
    The point of this whole thing is to get the pages cached by google and the se's.
    Basically All I need to know is, what user agents to check for.

    Thanks, hopefully this makes sence...

    Red.
     
    rederick, Dec 30, 2005 IP
  4. mad4

    mad4 Peon

    Messages:
    6,986
    Likes Received:
    493
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Check your server logs for the spiders that visit your site or visit this link for a full list.
     
    mad4, Dec 30, 2005 IP