I have a lot of direct visitors coming to my site too. I couldn't understand why at the beginning too
You most likely hit the nail on the head. Either javascript is disabled, or the person visiting has the referred-by tracking turned off.