A Members Only Section: Can Spiders Crawl it?

Discussion in 'Content Management' started by Litwriter21, Mar 13, 2008.

0
  1. #1
    What if the website has a members only section, can the spiders crawl that type of information, since they cannot log in?
     
    Litwriter21, Mar 13, 2008 IP
  2. Pixelrage

    Pixelrage Peon

    Messages:
    5,083
    Likes Received:
    128
    Best Answers:
    0
    Trophy Points:
    0
    #2
    It's possible - be sure to block it in your robots.txt:

     
    Pixelrage, Mar 13, 2008 IP
  3. Litwriter21

    Litwriter21 Peon

    Messages:
    19
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Thanks, for the reply.


    Currently we don't have a robots text. LOL


    For the sake of my non web developing brain, how would blocking access allow access? I'm soooo confused. :)
     
    Litwriter21, Mar 16, 2008 IP
  4. Tearabite

    Tearabite Prominent Member

    Messages:
    4,629
    Likes Received:
    429
    Best Answers:
    0
    Trophy Points:
    300
    #4
    my site has several members-only/premium areas, when the bots hit it, all they get is 'access denied'..
     
    Tearabite, Mar 16, 2008 IP
  5. Dan Schulz

    Dan Schulz Peon

    Messages:
    6,032
    Likes Received:
    436
    Best Answers:
    0
    Trophy Points:
    0
    #5
    No, it's not possible. If a person has to register/log in in order to access content, a search engine spider cannot access the content since they would need to register/log in, which they are incapable of doing.
     
    Dan Schulz, Mar 17, 2008 IP