Is there a tool or way to determine if a site was penalized by google? does de-indexed is a sign of penalty?
Definitely, De-indexing of a site from the search engine is a sign of penalizing. I think iwebtool has a tool to determine whether a site is banned or not.
If you don't have top ranking when searching for just domainname you've been SERP penalized. Generally you should be #1 for any search for your own name. For example if your domain name is stupidgoogle.com you should be #1 when searching for stupidgoogle.