Honestly, and I say this as someone who lobbies for [img] tags and goes to many different internet forums with different software, I don’t think there’s a much better alternative to vBulletin for a community forum. There are certainly a few tune-ups here and there that can be made, but vBulletin is really the best I’ve seen, and I’ve seen pretty much every forum software out there. That is, I don’t mean “vBulletin sucks, but it’s the best we have.” I mean “I can’t really think of many improvements I’d make to it.”
Most communities that have features are usually tied to the site (for instance Gaia Online has embedding of certain things, but those facilitate the game the site is based around). Certainly there are really good “community site” pieces of software that aren’t vBulletin, such as the Reddit system, Stack Exchange, imageboards, etc, but all of those serve the specific needs of the community. Stack Exchange is very handy for answering questions since it bubbles up good answers. Reddit’s system is very good for a site dedicated to recommending links on the internet, imageboards are good for very ephemeral, fast moving content based around images. vBulletin, likewise, is very good at what it does – discussion about random topics.
Perhaps the forum software of the future could have “environments” for different forums, for instance General Questions could have a Stack Exchange layout, MPSIMS could have a Reddit Style (while the “I’m sharing my life story” threads could be moved to IMHO), and the other forums could inherit a vBulletin-like theme. Ideally it would also have a button to convert between modes on the fly, so if I get tired of following the Reddit-style thread I could swap to vBulletin mode, same posts, but in chronological instead of rating/threaded order – but that may not work well for various reasons.
That said, there’s one feature I’ve only seen on one board (the old Nintendo boards) that I miss dearly, it was called a “float.” It was basically a user-sticky. You could click the float button and it would always appear at the top of the first page like a moderator sticky would, but just for you.
Edit: Also, a better search feature would be nice. At the very least a built-in option to redirect on search timeout. So if the forum would normally say “wait 30 seconds” it would search google instead (or at least offer to).