If no directive is given in the robots.txt file, do robots index password protected directories? Thanks for your input. Shannon
Thanks. That is good news indeed and just what I hoped. It will give a second layer of protection to keep content from being indexed. Shannon
Some search engines, notably Google, may still include pages in password protected directories in the SERPs, but without titles, descriptions or cached pages, if they find enough IBLs to determine what the protected page is about. They appear in the serps as the URL to the page.
Thanks, Bernard. I have disallowed spidering of the directory in robots.txt file but read somewhere that sometime information dissallowed in this manner may be indexed. Shannon
Yes, but, indexed <> cached. Pages or directories disallowed by robots.txt will not be read by robots.txt compliant spiders (such as Googlebot, Slurp, MSN Bot, etc.), so the respective search engines will not process or cache the disallowed content. However, the search engines may still include the URLs in SERPs if enough IBLs use meaningful anchor text. To ensure zero visibility of a page in the SERPs, you need to allow the pages/directories in robots.txt and use a meta robots = noindex tag. The meta robots directive does no good if the robots can't process it (because they were forbidden from reading the page).