Username,Password for crawler to index the site?

I recently opened my new site and you need to have a username and password to read the content.

I couldn't find anything when I looked at my "webmaster tool" to write username and password for google crawler to read and index the content at the site. I remember this (maybe 1 year ago I saw it or read it somewhere) but I just couldn't remember how do it right now.

Thanks

 

i think you are talking about Adsense Bot ? needs login information to login like any user and read content?

 

I was not aware that such a facility is available as never witnessed anything like it... myself. Just wondering...if it was possible to allow bot to index by giving it pass, how would it list your content in the serps for masses to read.. as you have reserved the content for privileged visitors?

Regards,

RightMan

 

Yeah, that's the only one I've seen. So the AdSense bot can target ads based on the content.
Trying to get Googlebot to index content that's behind a wall, will be hard unless you somehow cloak the content (not recommended). You could try allowing people to access part of the content without the need to login, then have a read more link to the rest of the content that's behind the wall.

Cheers
James

 

The only way you can find your password is to use of "Forgot my Password" link which is available in all websites which need login including Google Adsene or ... .
And if you can't remember your password by using it, you can not use of your account any time.

 

This would be cloaking. Cloaking = Penalized by Google.

 

OK guys' I got your points and it sounds understandable.

BUT somehow I know some forums(you need to register to able to see the content) and content is indexed by google.

PS: I don't use any torrent site and instead using rapidshare. So when I search as "blabla rapidshare" these pages are soming up at google. I checked their keywords and titles but it doesn't include these keywords. AND I can also see the links (www.rapidshare.com/file/blablabla....) at the search page so for sure it's indexed by Google. I don't know how but there is a way to do this (or google uses adsense bot user information)

 

have you read first page free?
Google accepts to give the bot a user and a password for 1 condition:
if the user found the page via google, he will be able to see the contents, then the website will use a cookies to disable this user from any other visit to any other restricted page even if he find it by google.

 

I think there's no way to index your site content if it needs password and username. Like what other said it can be done by cloaking. but it's unethical technique and search engine might penalized you.

 

for those that think it is risky.. what about nytimes.. wallstreejournal that are all behind pay-wall... what you need to do is on your server, make your content readable if client IP is from search engines.. for regular user.. they have to go through log in process... as long as you present the same content.. it will not be cloaking.. it is doable.. it takes a bit programming work.. thats all

 

IP? do you know how many IPs used by search engines?

 

just mail to Google services and help centre

 

webbmaster, your comment makes sense. Google has different IP's but there are some common IP's and these are the ones which are making indexing.
Depending on IP letting the visitor(bot) passing the login gate can make the site slower (u need to check every visitor IP) but it's worth if your content indexed.
I don't think it's a penalty from Google site but I'm not sure!