I wonder if there is anything left to SEO optimization when I use phpBB and Joomla. Do I need to make it anything modified or customize or the system is OK? What are the aspects I need to take care to make my site and forum optimized for search engines?
Few 'out of the box' forum scripts are fully search engine optimized. Many of them pass session ids through links which is bad for seo. Some also use dynamic urls instead of search engine friendly html urls. On page optimization isn't really the most important thing. As long as the search engines can index all of your pages, you should still be able to get decent rankings. It's just a matter of getting enough quality incoming links.
I think you get a lot of work to do, but you must stick to the basics : meta tags, content, back links. If you were using VBulletin it would much different as there is the vbSeo tool.
I agree with both answers. Just stick to the basics and focus more on your off page SEO since there is alot more involved in that. You should configure your metatags and content as best as possible and then put your energy into building high value links to improve your rankings. Good Luck!