I used to be the same but late have started concidering karma and now when I find a site(small to medium) with something usefull, I exit by clicking and Adsense ad.
I'm sure that if you click the ads frequently Google will pay the publishers less per click or even won't count the click because they think you're spamming the ads. Ofcourse if it only happens once a week it won't hurt.. I guess.
I agree with a comment earlier that the visitors who use adblock or other blockers are the type of visitors who are less likely to click on an ad if there was no ad blockers available. There are still the other type of visitors who are more likely to click ads which forms the majority.