Depends on your linux knowledge... for instance you can manage a Windows server pretty much just fine without a panel for it, you can install security on it, webserver, php, mysql, etc... same is with Linux... if you know how to do all that in Linux then it should be pretty easy to manage it without cPanel... if you don't know linux at all then... don't bother.
It really depends on what the server is going to be used for. I do agree with the statement above as well. If you just have one account, and don't plan on setting any more up, then it can be done. It might take a while to set things up properly, but, it can be done. A control panel automates many of the things that need to be done to set up accounts and so forth, along with some basic security measures.
You can try to start linux on some home virtual machine and get some knowledge. It's not so hard as you may think.
It's not so hard to install these applications and set up web sites - there are tons of guides available on the internet - but doing this to face a hostile internet environment is another thing altogether. It isn't just about installing these applications so they will work, but about installing them securely, making sure the server is secure, isolating web sites from one another so if one if compromized due to a problem script it doesn't affect other sites or the server, and maintaining the server so security and reliability are always optimized. These forums are full of posts from people whose sites or servers have been hacked, and if you have no server experience and aren't a systems admin, you will likely join these other posts in the future. Good server admins are hard to find, and they can spend a lot of time studying and learning about server security and trouble-shooting. You can't dismiss the amount of hard work it takes to be a competent admin. In my time I've met many people who can easily install applications, but when it comes to maintaining a server they haven't got a clue. Don't forget, cPanel isn't a security programme, it is only an interface that makes certain tasks easier. An insecure server is insecure whether it has cPanel installed or not, and if your server isn't secure, you are putting your data at risk.
You can't separate security from "management". There's no point telling someone it's easy to manage application installation then have them come back hours later saying their server has been hacked because they didn't think about security. Almost anyone can install an application no matter how inexperienced they are, but when it comes to managing a server then you have to be aware that when it's an internet-facing system security becomes an integral part of the management. Since he asked about how easy it was to manage one then I think my answer was a fair warning.
Cpanel would definitely make it much easier to manage a VPS or dedicated server, but if you have rhce equivalent knowledge, it's certainly possible to do it without Cpanel.