Allow googlebot to my members areas

Discussion in 'PHP' started by chickenhouse, Aug 23, 2008.

0
  1. #1
    Hi,

    I currently have a website with members areas that check for a userid. If there is no user ID they get redirected to the signup.php page.

    This works well in keeping out people but I want google bot and other bots to crawl these pages.

    Can I identify google somehow and use an if statement to allow google to these pages, or is there another approach I can use?

    I would be very grateful for any help on this.

    Thanks,
    Travis
     
    chickenhouse, Aug 23, 2008 IP
  2. NatalicWolf

    NatalicWolf Peon

    Messages:
    262
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    0
    #2
    User agent i believe...That would be the best way of doing it. But I don't recommend them being able to spider it...Thats risky sometimes.
     
    NatalicWolf, Aug 23, 2008 IP
  3. nico_swd

    nico_swd Prominent Member

    Messages:
    4,153
    Likes Received:
    344
    Best Answers:
    18
    Trophy Points:
    375
    #3
    The risky thing is that I could change my user agent to "Googlebot/2.1" (which takes about 2 seconds on Firefox), and voila, I'd be one of them, and could browse your page freely. So don't rely on the user agent. If you want to rely on anything at all, you might want to keep track of their IP addresses and work with these instead.
     
    nico_swd, Aug 24, 2008 IP
  4. ash84

    ash84 Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    you could change your authentication to http
    and theres an option for that in google adsense i think
     
    ash84, Aug 24, 2008 IP
  5. Steve136

    Steve136 Peon

    Messages:
    240
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #5
    One method i can think of is checking the visitors IP address and hostname using gethostbyaddr(); to check that it belongs to google.

    More information from PHP.net here,
    uk.php.net/manual/en/function.gethostbyaddr.php

    Regards,

    Steve
     
    Steve136, Aug 24, 2008 IP
    rohan_shenoy likes this.
  6. NatalicWolf

    NatalicWolf Peon

    Messages:
    262
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    0
    #6
    As others said, use gethostbyaddr...although some servers might not allow it...so, maybe have an ip array
     
    NatalicWolf, Aug 24, 2008 IP
  7. blackonyx

    blackonyx Peon

    Messages:
    223
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #7
    If you allow the Google bot to crawl your members area, then you risk your paid content ending up in the Google cache (don't forget the "rel=nocache" tag). Also Google is not very fond of getting sites presented in a different way from what the anonymous visitor would see. For the user it is a bad user experience, if a SERP shows a highly relevant page, which on click only leads to a sign up form. In fact, what you want to do can be considered to be "cloaking", which is against Googles webmaster guidelines and therefore something to not do:

    http://www.google.com/support/webmasters/bin/answer.py?answer=35769


    What you probably want is a CMS with it's own search function.
     
    blackonyx, Aug 24, 2008 IP