I have a content which is for registered users only. But I want the crawlers to see that content and not being directed to the login page. Also can you please list down the links of authority websites doing the same.
Make the content publicly available. It's as simple as that. If you still want to keep the good stuff available only to registered users while still having it found by the search engines, write summaries of that content and make the full version available only to your members. Remember, if a search engine encounters a "registration" or "login" form to access any content, they will not be able to see that content, meaning as far as they're concerned it doesn't exist.
why cant I check i/ps of the crawler and skip the login page and give them the content page. I dont think this would be cloaking or any malicious redirect issue. Will i be penalized if i do this ??
Because if the search engine spider has to "log in" to view the content, it will be blocked from seeing the content. It's simply the way some sites work. Either the content is available for everyone or it's only available for a few. You can't have it both ways unless you do what I suggested (either make it available or write a keyword-rich summary of what the content is about, and then have people log in or register to read the full version). And since the search engines will never see the full version (due to the login/registration form blocking them) they'll never know the full version even exists, thus saving you from a duplicated content penalty. And yes, the alternative I suggested is legitimate.
First compile a list of crawlers (user agents) you would like to index your pages. On your pages just check to see if the user agent is in the list, if it is don't redirect them to the login page. People can trick this by modifying their user agent, but its a simple solution.
It is possible via cloaking but certainly not recommended. WebmasterWorld used to do this until Google stepped in. Follow Dan's advice. - Matt
Not only that but my suggestion will create additional value - write summaries of content that you want registered members to read, and you might convert people who wish to "learn more" about what you had to say. Especially if the summaries were written with that goal (getting more people to register to access the premium content) in mind.
makes sense, thank you guyz. I asked the same thing at other forum and some1 told me if I check i/p address of crawler and redirect them I would be breaking google's guideline. Is it true??
Look at it this way - you're trying to serve a search engine spider content that is different than what a regular person would see. That's why it's such a bad idea (note: image substitution techniques using CSS to present a graphical version of a text block, such as with a header or menu item, is considered acceptable because the same content is available to the user if they are using a screen reader or happen to have images turned off - just in case some nutjob whacko says otherwise in the future).
http://code.google.com/apis/searchappliance/documentation/46/help_mini/crawl_access.html this thing has confused me completely