Do you mean that sitemap.xml is not being found or that pages that are referred to in the sitemap.xml are not being found?
I've been doing some digging for you and a sitemap with occasional 404 pages, will not affect your ranking. It is, however, preferential to keep the sitemap as up-to-date as possible - you would be a better netizen as the google db will have less 404 pages! S
You can bet there will be tons of old, discarded, non-updated sitemap files out there in the future. Google will count URLs in sitemap files like any other link from your site, so don't worry too much . However, you are loosing potential visitors by having URLs 404 that might be indexed and listed on the search engines. At the very least you should be using a custom 404 page (that returns 404) with a short site-menu on it to help the visitor get "somewhere".
I had a couple of 404's and it did not seem to affect my ranking. Since you know that there are 404's fix your sitemap and resubmit it.
Google have told on their words that sitemap doesn`t affect PageRank and their normal crawl system.. So don`t worry about that..
Oh, but if your robots.txt 404's then anything you have in there will / can be ignored. I wouldn't say that doesn't matter
I don`t know about it affects the ranking.. But it is not a bigger work to have 404 errors pages. It is easier with .htaccess..