Hello all, It seems to be funny when everyone try to get index in Google... but I want a different thing. I have a vBulletin forum and I would like to hide it from Google and other SE. Do you think it is possible? Thanks
rules for .htaccess may be difficult because you have hundreds of bots and bot - IP to consider for ALL SEs. robots.txt ONLY is respected by those wanting to respect the disallow rule. it is NO control nor security to keep all out. however the most simple way of keeping ALL world out - because that is what you most likely want, is to require login on the very root level of the site and thus to keep ALL world out except the ones authenticated.
An alternate method is to go to http://www.google.com/webmasters/tools/ prove that the website is yours (by uploading a trivial html file to the root of the domain), and then telling google to not spider your site, not index it, or remove it entirely.
Robots.txt is a joke. The search engines will still crawl the site even though you have a robots.txt telling them not too. They really don't care about it anymore.
Normally Google and other Search Engines can't index your site without backlinks or they won't be able to find it, so obviously if its a private website you won't be linking to it, so it might not get indexed anyway. Am I right?
There are cases even a site didn't get any backlinks still found by search engines. It's the contents the SE love.
The only way I know which works 99% time is to put password restriction on root of the site using .htaccess and .htpasswd and require valid user in .htaccess (which will pop up little box asking username/password).
follow these 1. put robots.txt disallow all 2. do HTACCESS password protect your site 3. go Cpanel put password on your /public_html/ folder
You can use any of this to deny access to the entire site/folder using: 1) robots.txt 2) .htaccess 3) meta tags: ex, nofollow,noindex,noarchive 4) password protected pages, via script/code
Google will find your site and sometimes ignore a robots txt but password restriction is the best case senerio. I just hope you arent hiding site because it has illegal content and you dont want to get caught... laterz malcolm
It should be understood that if you are connected to the internet, then anyone will see you information, site, etc. Even if you use a robots.txt file concept to block it, most of the second tier SEs will ignore it. The only way I have found is to have a shell website that works and is indexable and the real site within it but with no links what ever to it. Even then, if someone links to a page it will be indexed. None of the Meta scenario work either. And just for fun, even if you use a Google, Yahoo or MSN black hat SEO banned site, it will be indexed by the second tier SEs. And if this is not bad enough, the "Scrapers" will get you.
unfortunately the absolute ONLY ossible reason to completely hide a site is to do monkey business or worst type activities. any honest and clean activity always lives and prospers from public access and public reference / backlinks / being found. however as a hidden site your chance of being 24/7/365 on a global/national watch list and being observed by authorities as well as being attacked by hackers certainly is infinitely greater than publicly accessible sites. since all servers are physically located within US legislation - all server activities also may be monitored bypassing search engines! A site offering ONLY non-shipping products .... is vulnerable to sell pirate ware and alike ( or even worst ) .... just like so many sites in china, etc do ... you may rest assured that your HOST may carefully monitor your CONTENT unless your host loves to loose his business