Possibly encouraging precedent: The first Austin Powers movie didn’t do particularly well in its original theatrical run. Then it became an absolutely monstrous hit on video as word spread, which enabled the sequel. Interestingly, the sequel made in its opening weekend what the first film made in its entire original run.
I have to say, though, that having watched the machinations of Hollywood for a number of years, I’ve become pretty cynical about how the bean-counters make their decisions. If we get another Firefly movie, it will be after Whedon does Goners and Wonder Woman, and it will take both of those movies being extremely profitable (plus Serenity doing exceedingly well on video) for Whedon to have the clout to get it off the ground. I’d say odds are fairly heavily against.
Oh, and regarding all the plot nitpicking: Whedon’s narrative structure for the film is a fairly radical departure from the TV series. Whereas the show was largely character-driven, with long, leisurely storylines being played out, the movie is quite consciously built on a basic action-movie architecture. It is specifically and deliberately designed to rush past at a gallop, giving you no time to think, carrying you along with it. There’s really no wonder about why some of the plot points don’t exactly stick together in retrospect.
Example: What, precisely, was Jayne going to do when he went into the dining area where River was tied up? He has his weapon handy, and he mentions something about getting off the ship. Was he going to toss her out the airlock? Shoot her? Stick her in a shuttle and go collect the ree-ward? Or what? …Don’t bother thinking about it, because the answer is simple: Joss had a pretty big lump of exposition to deliver, i.e. River telling us what Miranda is about, and it would be boring if she just knocked and gave a little speech about it. He decided to layer the exposition in with an action-movie beat, so he had Jayne do something only vaguely motivated, and then launched into a cat-and-mouse game, culminating with River pointing a gun at Mal while pointing at the computer screen. In retrospect, those couple of minutes are completely unnecessary; they’re empty action-movie calories that don’t need to be there at all. It’s a spoonful of sugar with a droplet of expositional medicine on top. It’s entertaining while it’s happening, but in retrospect it’s kind of dopey.)
And the point is, the whole movie is designed to work like that. I know Whedon fans are sort of compulsive about revisiting material and looking for connections, and I’ll admit, there’s more here than in the average action movie that merits repeated viewings and detailed consideration. (Take the scene where River has her freakout on Miranda; the camera is whirling around her to suggest her confusion, and then Jayne steps forward, agreeing with her while the camera continues to circle. At the end, he says something like, “…and they all died for no reason.” At that point, the camera stops— with Wash in the center of the frame.) But I gotta tell ya, the movie just isn’t built the same way as the series, and trying to shoehorn all the bits and pieces together into a streamlined and sensible whole is a fool’s game.
Whedon, in a deliberate attempt to expand the film’s audience beyond the fan base, consciously made Serenity as an action movie, with all that implies. It isn’t supposed to inspire deep thinking, you guys. 