It’s actually not that complicated if you suspend disbelief long enough to accept that humans are a viable energy source. “Second Genesis” from the Animatrix explains what happened (some of that’s in the comics, too) to lead to the darkening of the sky. When that happened, the machines rounded up all the humans and began using them for energy. However, the humans died when their brains were unstimulated. So the machines created a Matrix.
Possibly the first one was a human paradise like Smith thought, possibly not. Either way, the humans ultimately rejected it and it crashed. The Architect and the Oracle, working together, figured out that humans would reject a Matrix, even a utopian one, because of the human abhorrence of control. So they built a new Matrix which gave humans – but only the dangerous humans – a choice. They could live within the Matrix or they could leave it and live in the real world.
Most humans are aware of this choice only subconsciously and happily live within the Matrix. When a given human is close enough to choosing differently such that he could threaten the stability of the Matrix, the machines arrange for the “free” humans from Zion to find him and offer him the red or blue pill. This is simply a symbol within the Matrix by which the target officially chooses the Matrix or the real world.
The machines see this as a necessary evil for the Matrix. They don’t want people taking the red pill, and they don’t like humans having too much power anyway, so they try to destroy any humans who hack into the Matrix, while at the same time allowing Zion to exist as a haven for those humans who cannot be safely contained within the program.
Eventually, Zion gets too big to control. The machines then raze it. At the same time, they use The One as another mechanism of control. The machines nurture a human (or maybe he isn’t really a human but a program implanted on a human’s brain) to be The One. Eventually they ensure that The One makes it to Zion. Then, while Zion is razed, the free humans who think The One is their messiah (instead of just another machine plot) get him in position to meet the Architect. While there, The One must make a choice. The Architect offers two options – either The One can maintain the status quo, Zion will be razed and all the free humans killed, but the Matrix’s 6 billion souls will continue and The One and a dozen companions will repopulate in a new Zion. Or he can refuse the status quo, the machines will still raze Zion, but they’ll also execute the billions within the Matrix. This is a bad deal for the machines, losing their power source, but they are willing to accept it if necessary. Allowing The One to have such a powerful choice is a necessary part of the revised choice-based Matrix and the only way it can survive at all.
Typically, The One has seen his options and accepted the lesser evil so as to avoid the extinction of humanity. However, in Neo his love and respeect for humanity is not generalized like the previous Ones, but instead is focused on Trinity, who is at that moment being killed by an Agent. Therefore, instead of taking the choice the previous six Ones have made and accepting the lesser evil, Neo refuses to play the game to save Trinity, hoping that he might figure out some way to avoid the extinction of humanity ad hoc.
Fortunately for Neo, the chaos that Smith is creating in the Matrix makes it impossible for the machines to delete it immediately. Eventually the machines and Neo make a deal. See, the machines made The One into the ideal Matrix hacker in order to give him power within the free human community and to ensure that he’ll be successful in reaching the Architect’s room at the appropriate time. (It may also be a side-effect of whatever processes the machines do to make a person The One.) Therfore, he is in a position better than any of the machines to fight Smith. Smith is essentially a virus in the Matrix, ergo, something outside the tainted main program needs to combat him.
Neo is successful in eradicating the Smith virus from the Matrix. Therefore, his deal with the machines goes into effect. They will allow Zion to continue to grow in peace. They will continue to operate the Matrix, but every human will be presented with the choice, and anyone who wants to leave the Matrix will be allowed to, instead of limiting it only to those necessary to bleed off the pressure on the program. This isn’t ideal for the machines because it loses them some fraction of their batteries and, more importantly, it lets Zion grow and become more sophisticated. The machines know the truce won’t last forever, and they don’t want to be in a position to have to sweat the fight with Zion when it collapses. But they will honor the deal.
As for the humans, some of them will leave the Matrix. But most of us won’t. On the literal level, that’s the ultimate point of the story – there’s nothing intrinsically better about living in the “real” world than in the Matrix. Life is life. When you’re in the Matrix, there’s nothing about it that’s less moral or less satisfying than experiences of the real world would be, because it’s just as real. Cypher was a bastard who betrayed his friends, but there was nothing immoral about his ultimate desire to be back in the Matrix. The important things in life are about our relationships and being fully actualized. It doesn’t matter a whit how many levels you are removed from bedrock reality as long as you’re successful within your stratum. The first film asked the question “What if none of it was real?” The trilogy as a whole answers that question – and the answer is that is doesn’t matter whether it’s real or not; that is, life is real even if the world we see around us is fake.
–Cliffy