I set up a test this week for Adwords at work...all I did was drop my account off of the Search Network. As a result, the average daily cost dropped from approx. $260 to approx. $170, while clicks and impressions remained the same, and conversions actually rose by a small amount. Through Analytics, I noticed a drop in educational & ISP visits, and an increase in corporate visits - which is exactly what we were looking to achieve. In short, if you're looking for a way to optimize your campaign, set your account to advertise only on Google.com (and not the Search network), and try it out for a few days. This probably works only for certain niches - ours is business training. Hopefully it will reduce the amount of worthless clicks you have been paying for!
I've always wished Google would let advertisers test the search network as a stand alone segment like they do the content network. For the most part I see exactly what you do - a drop in performance when the search network is on.