Try to disable 3 modules: overlay, toolbar and dashboard, and just use the admin menu module. That is my setup right now, or the your web server can't handle Drupal 7. Panel is my next lesson, modifying my front page and it's looks..
I have dashboard and overlay off but the toolbar on I actually managed to get *one* of my d7 sites smaller by removing the javascript. It didn't need it but includes it unconditionally. Had to stop it rendering script html tags in the theme, which is a dodgy fix and only works on sites that don't need javascript *anywhere*. The site is now 66k including all css and images, and very fast. Webmaster tools reports an average 164ms load time, which is way under my larger d6 sites.