what will happen if the users find a thread in the serps and click on it? Will they be able to see the posts or will it take them to the signup page on my forum?
I don't know this for a fact, but I would think they would be directed to your signup page - anything else would seem to defeat the purpose of having a private forum. You may want to test with another forum you know of that's private (and listed in the SEs), as this would probably be a good indicator of what will happen with your forum. HTH, Sam
Assuming that your forum's posts will even show up in the SERPs, the users will get what ever you get when you try to view a thread without being logged in. However, the crawlers will get this same thing (since they are not logged in) thus these pages will never be indexed in the first place (unless they were indexed at a point when login was not required)
what's the point of having member only access and simultaneously relying on search engines to bring traffic?
Agreed, I would think it takes them to registration as well. Although, you have to do LOADS of modding your Forum to have Search Engines even look at it at all, not to mention index it.