1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Can I restrict access to certain pages with robots.txt?

Discussion in 'Search Engine Optimization' started by hello2, Jun 18, 2011.

  1. #1
    Is there anything wrong with restricting some access through the robots.txt file (for example, the privacy policy)?
    Will this impact the SEO ? Will search engines think I'm hiding content?
     
    hello2, Jun 18, 2011 IP
  2. Vineet Koushik

    Vineet Koushik Member

    Messages:
    61
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #2
    Robots.txt. can restrict only search engines crawler to index your content
     
    Vineet Koushik, Jun 18, 2011 IP
  3. hello2

    hello2 Greenhorn

    Messages:
    17
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #3
    you mean there's something else that can index my pages?
     
    hello2, Jun 18, 2011 IP
  4. revium

    revium Member

    Messages:
    330
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    33
    #4
    I think what Vineet Koushik means is that there are some bots and crawlers (none of the major search engine ones though) that do not obey the directives in the robots.txt file. So in those cases there is no way to stop them from indexing that page unless you block their IP.

    So for your case, yes you can block a specific file in the robots.txt from the search engines without any problems. I doubt the search engines will think you are trying to do something devious or hide anything.

    You can also add a noindex,nofollow meta tag to individual pages you want to block from them. You can do this instead of making an entry in the robots.txt file. The meta tag comes in handy if you do not have access to the robots.txt file. Also the robots.txt file is useful for blocking entire sections from the search engines, such as an admin section, if you do not want to go through the trouble of adding the noindex,nofollow meta tag to each individual page.

    Out of curiosity, why do you want to restrict them from seeing your privacy policy page?
     
    revium, Jun 18, 2011 IP
  5. dcristo

    dcristo Illustrious Member

    Messages:
    17,997
    Likes Received:
    881
    Best Answers:
    2
    Trophy Points:
    420
    #5
    Yes, like revium suggested you can block specific pages/folders in robots.txt
     
    dcristo, Jun 18, 2011 IP
  6. hello2

    hello2 Greenhorn

    Messages:
    17
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #6
    Because I have that on other websites, same text, so it would look like duplicate content. That goes for the "About me" page as well.. and others
     
    hello2, Jun 18, 2011 IP
  7. revium

    revium Member

    Messages:
    330
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    33
    #7
    I do not think you need to worry about duplicate content in your privacy policy page, most sites use boiler plate text or similar text, but if it makes you feel more comfortable then go for it.

    Personally I would not worry about blocking those pages. Google is not going to penalize your whole site for just similar Privacy Policy and About Us pages.
     
    revium, Jun 18, 2011 IP
  8. dcristo

    dcristo Illustrious Member

    Messages:
    17,997
    Likes Received:
    881
    Best Answers:
    2
    Trophy Points:
    420
    #8
    Do you know how many sites have the same privacy policy, disclaimer pages, etc? You won't get penalized for it, google is smarter then you think.
     
    dcristo, Jun 18, 2011 IP
  9. hello2

    hello2 Greenhorn

    Messages:
    17
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #9
    But why risk it? Anyways, I've already done that and since it's not a problem for SEO, it's perfect for me. Thanks all!
     
    hello2, Jun 18, 2011 IP