This Tutorial works only with Mozilla Firefox A lot of different websites allows the Google Robots (G-Bot) to view and analyze their site. So from this, forums and websites allows him to crawl the site, without a need to register or login. So if you also want to visit sites like the Google Bot, you need to do this: First of you need to install this Addon: UserAgentSwitcher After you installed the Addon, go right here: Tools>User Agent Switcher>Options>Options Klick on: "User Agents" and then on "Add" Fill out these things: * Description: Googlebot * User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) * App Name: Googlebot * App Version: 5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) * Platform: http://www.google.com/bot.html * Vendor: -this on you leave that * Vendor Sub: -this on you leave that In the end klick on OK! So now on 'Tools>User Agent Switcher' click on 'Tick GoogleBot' And now every site will think, that you are a Google Bot. Go and try it out Sometimes you also have access to VIP sections, maybe you will also find one. Greets Albaniax
this does actually work, but only if the site in question isn't using bot authentication. And sites that (eg) use cookies to log in and out still wont let you in the private areas.
Switching the user agent is easy and therefore known by many webmasters. GG and other bots can be identified by their IPs (or rather group of IPs )
as I've already said, the bot authentication routines recommended by the se's themselves are less than perfect (especially MSN). This means that the risk of a 'false positive' is high, and so many sites decide it's not worth the risk. After all, if you ban the 'real' bot by mistake, guess what, you aint getting crawled, sucker.