1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Is google indexing password protected directories/urls ?

Discussion in 'Search Engine Optimization' started by thesamemanhal, Dec 17, 2010.

  1. #1
    I have a very old backup copy of my website and decided to reinstall it on a subdirectory of my current updated website (I need the old site occasionally for some references).
    I have set a password protected to the subdirectory form cpanel so that no one can see it except me.
    To my surprise google indexed a page of this subdirectory even though it is protected! and that makes me worry because of the duplicated contents.
    Any clue about this ?
    SEMrush
     
    thesamemanhal, Dec 17, 2010 IP
    SEMrush
  2. TheCrabb

    TheCrabb Peon

    Messages:
    304
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #2
    It's best to exclude any pages you don't want found in your robot.txt file as well as adding "no index" tags to your pages.

    Google wouln't have spidered your page directly by sending google bot through your login form, but they could have accidently found it some other way such as unsecured sever logs or something.
     
    TheCrabb, Dec 17, 2010 IP
  3. yoes_san

    yoes_san Peon

    Messages:
    443
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Google will index password protected directory - but it can't index protected URLs.

    Because even if you protect the directory, people can still access it via URL - it's a different thing, directory and url.
     
    yoes_san, Dec 17, 2010 IP
  4. webdev007

    webdev007 Active Member

    Messages:
    1,037
    Likes Received:
    13
    Best Answers:
    3
    Trophy Points:
    88
    #4
    Yes, Robots.txt is one of the best ways to not allowing Google and other major crawlers to index unwanted or protected urls.
     
    webdev007, Dec 17, 2010 IP
  5. anna.frank

    anna.frank Peon

    Messages:
    56
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I agree with most of the experts above. Robots.txt is the best way to restrict google from crawling your web pages.
     
    anna.frank, Dec 17, 2010 IP