how to allow spiders to bypass login page

Discussion in 'Programming' started by MrPJH, Jan 7, 2011.

  1. #1
    i run a website where users must be logged in to view content and if a visitor click/visit on a content page from anywhere it redirects to login page but i want to allow spiders bypass this
    how to do this math
    my site is developed in PHP

    thanks
     
    MrPJH, Jan 7, 2011 IP
  2. shofstetter

    shofstetter Well-Known Member

    Messages:
    178
    Likes Received:
    7
    Best Answers:
    1
    Trophy Points:
    120
    #2
    you could check the user agent string:

    
         function check_if_spider()  
         {  
             // Add as many spiders you want in this array  
             $spiders    = array(  
                             'Googlebot', 'Yammybot', 'Openbot', 'Yahoo', 'Slurp', 'msnbot',  
                             'ia_archiver', 'Lycos', 'Scooter', 'AltaVista', 'Teoma', 'Gigabot',  
                             'Googlebot-Mobile'  
                         );  
       
             // Loop through each spider and check if it appears in  
             // the User Agent  
             foreach ($spiders as $spider)  
             {  
                 if (eregi($spider, $_SERVER['HTTP_USER_AGENT']))  
                 {  
                     return TRUE;  
                 }  
             }  
             return FALSE;  
         }  
    
    
    Code (markup):
     
    shofstetter, Jan 7, 2011 IP
  3. AstarothSolutions

    AstarothSolutions Peon

    Messages:
    2,680
    Likes Received:
    77
    Best Answers:
    0
    Trophy Points:
    0
    #3
    it depends how import spidering is -v- security agent is a good option for a low risk site but it isnt hard for users to fake agent responses and so potentially accessing content and likewise serving up different content may be considered Black hat
     
    AstarothSolutions, Jan 8, 2011 IP
  4. MrPJH

    MrPJH Well-Known Member

    Messages:
    1,066
    Likes Received:
    7
    Best Answers:
    1
    Trophy Points:
    155
    #4
    thanks its not a problem content is not pay to access so no problem
    actualy i run short task script and before when user click on a job/task system says login first in this way spiders also face the same and do not crawl even title of the task
    i tried a lot which one posted in #2 reply but dont work
     
    MrPJH, Jan 9, 2011 IP
  5. mikeasro

    mikeasro Peon

    Messages:
    145
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #5
    .... Its easy to fake a user agent, I would't use this method if you want the directory to remain secure
     
    mikeasro, Jan 11, 2011 IP
  6. MrPJH

    MrPJH Well-Known Member

    Messages:
    1,066
    Likes Received:
    7
    Best Answers:
    1
    Trophy Points:
    155
    #6
    signup is free
    it takes less then one minute to sign up then if some one fake user agent its not a problem i am just worrying that users can view the content because they can sign up but spiders dont
     
    MrPJH, Jan 11, 2011 IP
  7. sql

    sql Peon

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Maybe you can use cookies to detect if it is spiders, since most search engines don't accept any type of cookies.
     
    sql, Jan 11, 2011 IP