Need to hide from search engines

Discussion in 'Search Engine Optimization' started by rob7676, Aug 26, 2009.

  1. #1
    Yes, I know, not something you see every day. One of the company's I work for supplies some of their clients with website to store some data. These site need to be hidden from search engines. Is there a way to do this?
     
    rob7676, Aug 26, 2009 IP
  2. elenabush

    elenabush Peon

    Messages:
    502
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #2
    if site would be hidden personal data than how know to progress search engine.
     
    elenabush, Aug 26, 2009 IP
  3. rob7676

    rob7676 Peon

    Messages:
    82
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I don't quite understand what you said.
     
    rob7676, Aug 26, 2009 IP
  4. ipr22

    ipr22 Peon

    Messages:
    113
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Hide the data behind password protected areas. Or you can use robots.txt, but any normal user could just look at the robots.txt file to see where the data is in that case...
     
    ipr22, Aug 26, 2009 IP
  5. bonniecool

    bonniecool Peon

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    if it's a webpages don't make a link that will point to that datas or
    create robot.txt and disallow the folder that you want se robots not to crawl.
     
    bonniecool, Aug 26, 2009 IP
  6. rob7676

    rob7676 Peon

    Messages:
    82
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #6
    I've thought about the password protect, but I've been instructed to find a way to keep the clients url from even appearing in search engines.
     
    rob7676, Aug 26, 2009 IP
  7. pubdomainshost.com

    pubdomainshost.com Peon

    Messages:
    1,277
    Likes Received:
    21
    Best Answers:
    0
    Trophy Points:
    0
    #7
    robots.txt is the way to go. If Google/Bingoo doesn't index pages - it doesn't show them in results to any user and thus no one would know what all pages are turned off for spiders.

    The other option could be to use .htaccess to block access to bots and users too. You can control which IP address can see what on your server using .htaccess directives.
     
    pubdomainshost.com, Aug 26, 2009 IP
  8. rob7676

    rob7676 Peon

    Messages:
    82
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #8
    I will look into the robot text. Thanks
     
    rob7676, Aug 26, 2009 IP
  9. sherinjacob

    sherinjacob Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    The robots.txt protocol is used to tell search engines (Google, MSN, etc) which parts of a website should not be crawled.

    I think password protecting your the data is the best way.
     
    sherinjacob, Aug 26, 2009 IP
  10. farangkwaiyai

    farangkwaiyai Guest

    Messages:
    317
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Make sure you tell them not to crawl and not to index.

    Pages will still get indexed even if they are not crawled.
     
    farangkwaiyai, Aug 26, 2009 IP
  11. vividwebgraphics

    vividwebgraphics Peon

    Messages:
    45
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Use robots.txt file to hide any pages of your website.Even you can use meta robots tag to hide from search engines.
     
    vividwebgraphics, Aug 26, 2009 IP
  12. Amator

    Amator Well-Known Member

    Messages:
    1,424
    Likes Received:
    55
    Best Answers:
    0
    Trophy Points:
    165
    #12
    Amator, Aug 26, 2009 IP
  13. Natheen2000

    Natheen2000 Peon

    Messages:
    154
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #13
    You have to play around with robot.txt and disallow those web pages which you want to hide from search engines. :)
     
    Natheen2000, Aug 26, 2009 IP
  14. mousee

    mousee Active Member

    Messages:
    207
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #14
    robots.txt works really good. I had the same problem and this was the answer.
     
    mousee, Aug 26, 2009 IP
  15. cyberblogteam

    cyberblogteam Peon

    Messages:
    15
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #15
    The best thing you can use for your scenario would be using a .htaccess file. Here's the supporting video info for this:

    http://www.youtube.com/watch?v=nM2VDkXPt0I

    Matt Cutts explains why at 2:24 of the video.
     
    cyberblogteam, Aug 26, 2009 IP
  16. pageupseo

    pageupseo Peon

    Messages:
    770
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #16
    use robot.txt for hide you url from search engine.
     
    pageupseo, Aug 26, 2009 IP
  17. remshad

    remshad Active Member

    Messages:
    324
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    55
    #17
    Make a robots.txt and add nofollow to your links or folders
     
    remshad, Aug 26, 2009 IP
  18. f.anto78

    f.anto78 Peon

    Messages:
    37
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #18
    i think make meta tag at your side [ordinary at header]robot.txt noindex
     
    f.anto78, Aug 26, 2009 IP
  19. outrefranc

    outrefranc Active Member

    Messages:
    94
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    93
    #19
    robots.txt is not enough. It will tell robots not to go there, but will not prevent them to do so. To be safe, use a password.
     
    outrefranc, Aug 26, 2009 IP
  20. Shopplugin

    Shopplugin Peon

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #20
    The best sure-fire way is to restrict access to your site via .htaccess. You can put an IP restriction so only you or whoever else you want can view the site and not the search engines.
     
    Shopplugin, Aug 26, 2009 IP