Can someone damage your site's google SERPs?

If somebody had hurt your serps then it would be pretty easy to find out.

Visit yahoo and search for link:http://www.yoursite.com and see what links come up. If you get 2 million links with the anchor text "poker" then something has gone wrong.

 

Here's another thread with some interesting discussion on this:
http://forums.digitalpoint.com/showthread.php?t=229198

 

Your website is probably dancing with google, if your website jumping up, and down from the serps you need more link juice, so try to get some quality relevant links built up, and no one can really hurt your serps, unless someone wanted to take the time out and black hat your website, but other then that no one could hurt your serps, and if someone was black hatting your website google would either ban you, or take your page rank.