Most of the my blog commenter put there blog or website URL when they commenting. So if my blog has lots of these kind of links then Google will reduce my SEO?
Yepp, the amount of outbound links will increase and the page will be considered to be a spam page by Google which will ultimately decrease your pagerank.
Try to moderate your blog and keep only comments which are meaningful and relevant. I am sure you know about dofollow and nofollow as well.