Does anyone know if the spiders for the localized Google sites (co.uk, com.au etc) actually have IPs from those countries? Does the Google.com.au spider see my IP based geo targeted AU content, etc... Or do they all just crawl from the US?
I'm going to assume that they have IPs from those countries. I'm guessing this is why there are different serps for each individual google site. If everything was just spidered by a US spider, wouldn't all the rankings on the google.com, google.ca etc. be the same?
This is a pain in the ***! I have had this issue with google for as long as I can remember! I have GEO IP detection on my site that displays language and content dependent on the location and google.co.uk still get crawled by a US bot and so does google.de etc... I have had to create so many more pages to compensate for this ridiculous situation.
As far as I can ascertain, the servers for both Google.com.au and Google.co.uk are located in the US. So I'm kinda assuming that they probably send their spiders out from there too. @ Gazzerman: do you know for sure that the spiders are coming from the US? (If I could be bothered to check my logs, I'm sure I'd find their IPs) I guess if the AU spider has a fixed IP, then we could serve it up AU content... etc ??? I don't understand... ? How does this help?
All our pages that are indexed for the UK have the US information in the cached pages and the same for elsewhere. I have to create links to static pages for those languages just so that they get crawled, if google crawled from the correct country there would be no need to have a /german/info.html /french/info.html it would simply index the info.html in each language if the bot came from the correct location.