This may be posted already but i couldn't find it. How do you create a sitemap for a website that has password protected content. When you reach my homepage www.mvpexchange.com, users are prompted to enter a username and password. the crawlers are therefore only looking at this one page and another registration page. How do i give the crawlers access to my entire website? I have heard some things about robots.txt?? thanks, Jon
maybe you can change the content of the robots.txt which resides in the root directory of your website
If your content is protected from anonymous browsing, then there is no way to allow crawlers nor would you want them in there. Do you want Google to show pages that you are trying to hide from everyone except registered users? I don't think so. All you can do is have a rich home page that has samples of what is inside and a login/register box on it.
Well, it requires some configuration, but you can often have A1 Sitemap Generator sitemap protected pages. However, it rarely makes sense, but may do if you are thinking about 1-click-free solution like proposed by Google. (Also linked in article)