Passive traffic has a very low value overall. General websites with just semi-relevant to relevant ads is just not a high quality buyer and thus does not convert well for a lot of affiliate offers and also can lead to adsense smart pricing. Some of the advice here is spot on. If you want to maximize your revenue on your websites you will have to lather up your visitors and pre-sell them to get them in a purchasing mindset. Then you can make some money. Otherwise your level of traffic is substandard and worth less than other sources of traffic.
The key to affiliate marketing is to make the offers relevant. Adwords works only because it shows relevant ads. So if you can find relevant affiliate programs/offers, you would do fabulously. If you put wrong affiliate offers on your pages, you would lose out. It's all about trial and error. Trying something and if it does not work, trying something else.