Dont go with subdomain. Hard to maintain and google sees it as a new domain so you can't pass PR easily. Forum is a folder is a better approach than this but best is installing forum directly in the root folder if this is an option for you. It helps your threads to be closer to main hence will have SEO advantage.
If you're more concerned with usability and convenience of your members, go for subdomain. But if is SEO is your top priority, go for subfolder. If you were to ask me, I prefer subdomain just like here in DigitalPoint.
If you dont use many mods dont use a folder for the site. If you have a heavily modded site, its better to use a folder for when you upgrade. It makes it easier to work with when swaping files/folders after an upgrade.
Ditto...only use subdomain if you plan to section off your sites in a way that allows you to later on use seperate servers. Really using subdomains is best as a load balancing technique more than anything else. Such as DP which might be why they do
In my opinion, a subdomain appears the most professional... but that may not be your goal. So in the end it really is preference (as was said in an earlier post).
I've been wondering the same thing, but with a twist. I want to get a forum up and running, and in time have it be part of a larger cms based site. Do you all think it would be better to install it in a folder in this situation too? As an example, I'd really like to do something with drupal but it's not cast in stone; it could be any cms. But before i know enough about the chosen cms to launch a site with it, i plan on having the forum up and running.