1,000 unique visitors a day, let's say each one visits an average of 1.5 pages, that's 1,500 ad impressions per day. If you have an eCPM of $5 (for instance with CTR 5% and CPC $0.1) then your site is earning $7.5 a day. He gives you $3 so he's earning $4.5 good deal. As you can see, for a eCPM of $5 you need 20,000 ad impressions to earn your $100 a day. You should improve your traffic more than 10 times and have your own adsense on your site. $100 is way possible, the thing is getting enough traffic that's it. Also be aware that you can have several ad network on the same page with adsense, so you can earn $75 with adsense and $25 with other ad networks, that will still earn you $100 a day.
It could be that your site is earning more than the $3 a day and he is actually ripping you off. I do not understand why you will have someone else manage your adsense earning?????? It makes no sense to me at all. It could be possible that he also built other sites and used your adsense account to earn him a sizable monthly income. I don't know. It is all speculation on my part based on your post. The thing is that you can increase your earnings to even $100 a day (it wont happen overnight of course) but you must first have complete control over your account.