AFAIK they are using a slightly modified version of Ubuntu which is internally called Goobuntu. Some time ago it was rumoured that Google is going to release Goobunto but Google denied this.
I'm pretty sure the whole Goobuntu thing was just a desktop OS rumour, and doesn't have anything to do with their datacenters. From what I've read/heard, the datacenters run Google's own customized version of Linux. Source: http://www.internetnews.com/xSP/article.php/3487041 As far as webserver... I'm pretty sure they are using IIS (sarcastic)