While I could code up something myself, I'll point you to a site with some examples and codes. www.php.net/rmdir Have a look at the user comments.
You didn't really specify you want to do this with php I'll just tell you that you can login with your ftp client to your website and select the directory and press delete key, other than that I don't imagine a faster way.