i run a website where users must be logged in to view content and if a visitor click/visit on a content page from anywhere it redirects to login page but i want to allow spiders bypass this how to do this math my site is developed in PHP thanks
you could check the user agent string: function check_if_spider() { // Add as many spiders you want in this array $spiders = array( 'Googlebot', 'Yammybot', 'Openbot', 'Yahoo', 'Slurp', 'msnbot', 'ia_archiver', 'Lycos', 'Scooter', 'AltaVista', 'Teoma', 'Gigabot', 'Googlebot-Mobile' ); // Loop through each spider and check if it appears in // the User Agent foreach ($spiders as $spider) { if (eregi($spider, $_SERVER['HTTP_USER_AGENT'])) { return TRUE; } } return FALSE; } Code (markup):
it depends how import spidering is -v- security agent is a good option for a low risk site but it isnt hard for users to fake agent responses and so potentially accessing content and likewise serving up different content may be considered Black hat
thanks its not a problem content is not pay to access so no problem actualy i run short task script and before when user click on a job/task system says login first in this way spiders also face the same and do not crawl even title of the task i tried a lot which one posted in #2 reply but dont work
.... Its easy to fake a user agent, I would't use this method if you want the directory to remain secure
signup is free it takes less then one minute to sign up then if some one fake user agent its not a problem i am just worrying that users can view the content because they can sign up but spiders dont
Maybe you can use cookies to detect if it is spiders, since most search engines don't accept any type of cookies.