Do you want to compete with vbulletin specifically? In my opinion there are two basic factors: marketing anf features of your software. It is difficult to compete in terms of features. You may have an advantage for the web marketing if you know your niche well. But you should know your niche well, have some realted old sites and a good plan...
Shouldn't be hard. You need to either: Have enough money to fund advertising and promotion yourself or Know enough people who can do this for you. If you do the first option, make sure your product is good enough to get sales. Theres no point in spending £10,000 promoting, if your product isn't good, doesn't have features or won't sell.
Well, it is definitely not impossible. You will need to look for a niche here. Find what people wants and what vBulletin does not provide and work on that. Also look at the other Bulletin scripts out there as they can be your potential competitors.
It's very possible. I would suggest that you start writing your business plan. This is the second step after the genesis of the business idea. Writing plan for business is a very basic thing in business start up but most people do neglect this. Some think they already had all the ideas and resources to run the business. Some may succeed without a concrete plan but most of them failed. Business plan clears the cloud about your business visions. Don't be afraid about those giant software companies. They might be too big to compete with, but believe me, If you are willing to work hard and had acquired enough resources for the start-up phase,slowly and slowly you'll be rewarded with a market share that's right for the size of your company. Armed with quality products/services to offer, I must say, up you will go!