If people who bookmarked your site in IE or FF coming to your site daily and if they click on your ads for true reasons would Google notice this as click fraud too ? Like lets say you have a page or site about free coupons and people might come to site daily and they would click on ads which mention about free coupons or discounts.
No i do not think so its about what they do on the site as well how long time they spend there and what they are doing. If they only click on the link and leave every day so maybe yes there will be a problem.
i dont think that it will be a problem for google. it's a normal action for the nature of the internet