Well a nice way to prevent a lot of these dangerous shell scripts (r57, c99) is to disable the functions that power them to begin with. So something like disable_functions = exec,shell_exec,passthru,proc_open,proc_close,system covers a lot of them. Of course mod_security rules can prevent a lot as well but who knows maybe you have a malicious user? But really these scripts should do absolutely nothing to you if you run openbase_dir setup or have suPHP and such configured properly. Along with that up to date kernel so they can't use kernel based exploits to then have root privs so they can write to other directories.
May i get a copy as well? Is there a simple way to find out whether if there were rootkits uploaded ? But disabling those functions will also prevent some some common PHP scripts from running no ?
Very few scripts have uses for opening processes and executing server commands. The only ones I can think of are used to report system information by using the system command.
Google mod-security and install it. If you have root access and control panel is cPanel/WHM, this can be done by the "Plugins" section under "cPanel" tab in WHM.