Bread crumbs is a navigation aid. It helps user to keep track of where they are in the site. This is very true especially for huge content sites where they have hundreds or thousands of pages within various categories. Bread crumbs will therefore tells the user their location. Eg of how bread crumbs looks like : Sports - Basketball - Basketball Playing Tips (where Basketball is a category and Basketball Playing Tips is the article within that category) The search engine like Google favor bread crumbs as part of the on-page optimization. Well, as long as it enhance user experience, Google will love it as Google always put higher priority to user experience.
The answer above is spot on To add, this also has some excellent SEO value. as it also directs the bots back and forth the deep buried pages in a site and get them indexed. Breadcrumbs are an excellent way to get your site listed in Sitelinks
There are more WordPress breadcrumbs plugins http://wordpress.org/extend/plugins/search.php?q=breadcrumbs&sort=
Sounds tasty. When I decided to use a breadcrumb on my blog I ended having to try a few until I found one which worked. Depending on the theme you use I suggest you do the same, for example the yoast plugin gets good reviews but it didn't work for me. I spent a bit of time fiddling with it, but it was a waste of time in the end as I tried NavXT and that worked straight away.