Hi, I currently have a website with members areas that check for a userid. If there is no user ID they get redirected to the signup.php page. This works well in keeping out people but I want google bot and other bots to crawl these pages. Can I identify google somehow and use an if statement to allow google to these pages, or is there another approach I can use? I would be very grateful for any help on this. Thanks, Travis
User agent i believe...That would be the best way of doing it. But I don't recommend them being able to spider it...Thats risky sometimes.
The risky thing is that I could change my user agent to "Googlebot/2.1" (which takes about 2 seconds on Firefox), and voila, I'd be one of them, and could browse your page freely. So don't rely on the user agent. If you want to rely on anything at all, you might want to keep track of their IP addresses and work with these instead.
One method i can think of is checking the visitors IP address and hostname using gethostbyaddr(); to check that it belongs to google. More information from PHP.net here, uk.php.net/manual/en/function.gethostbyaddr.php Regards, Steve
As others said, use gethostbyaddr...although some servers might not allow it...so, maybe have an ip array
If you allow the Google bot to crawl your members area, then you risk your paid content ending up in the Google cache (don't forget the "rel=nocache" tag). Also Google is not very fond of getting sites presented in a different way from what the anonymous visitor would see. For the user it is a bad user experience, if a SERP shows a highly relevant page, which on click only leads to a sign up form. In fact, what you want to do can be considered to be "cloaking", which is against Googles webmaster guidelines and therefore something to not do: http://www.google.com/support/webmasters/bin/answer.py?answer=35769 What you probably want is a CMS with it's own search function.