Password-required forums and crawlers

Discussion in 'Search Engine Optimization' started by Kosmix5, Apr 6, 2009.

  1. #1
    Does backlink activity on forum pages that are only accessible by password have any benefical effect on SEO?

    And while we're on that subject, do "adult content" splash pages also limit crawler activity?

    Thanks-
     
    Kosmix5, Apr 6, 2009 IP
  2. Canonical

    Canonical Well-Known Member

    Messages:
    2,223
    Likes Received:
    141
    Best Answers:
    0
    Trophy Points:
    110
    #2
    If you have to login in order to see content, the crawlers will never see it. They are not going to register a login, enter a new username/password, and press "login" to get to content.

    If the adult content splash pages require you to login before proceeding to the site then yes it will limit the crawler.

    If the adult content splash pages simply say "Click Here if 18" and that link is a dofollow link, they will follow it.

    They would not follow the dofollowed "Click Here if 18" link if the page it is on or the target of the link has a <meta name="robots" content="noindex,nofollow"> or if the page or folder it is in is disallowed in your robots.txt.
     
    Canonical, Apr 6, 2009 IP
  3. waxman1000

    waxman1000 Peon

    Messages:
    1,365
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    0
    #3
    1 million backlinks on a page that requires a login is equal to zero backlinks cos the robots wont be able to crawl it!

    Google has been saying for quite a while now that they are working on that problem. maybe in the future Google crawl bots will be able to login and crawl those kind of pages, but as for now it is not possible.

    crawls do crawl adult content. that is why Google have adult sites in there database of websites to respond to those who want to view such sites.
     
    waxman1000, Apr 6, 2009 IP
  4. BlackhatVault

    BlackhatVault Banned

    Messages:
    262
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Do the website owner need to tell the user ID and passwords to google or what? Because i cannot see the bots crawling these log-in pages without knowing the passwords and ID.

    Still confused how can that happen..?
     
    BlackhatVault, Apr 6, 2009 IP
  5. Alightstudios.com

    Alightstudios.com Peon

    Messages:
    27
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Some sites will let a crawler in while blocking a regular visitor, its a form of cloaking but some large forums still do it. So google can read the site content and backlinks will be counted. Also if its user agent based you can access these sections by spoofing the google user agent, if its IP based, probably not.
    However in most cases if you need to log in,google cant get in.
     
    Alightstudios.com, Apr 6, 2009 IP