This isn’t really a question with a General Answer, but I can offer a little insight:
Microsoft doesn’t do a lot of innovating. Since time immemorial their strategy has been to take something good and make it better. This worked out very well for them in industries with established competitors (Word, Excel, Windows), but even ten years ago they were starting to worry about becoming victims of their own success.
Consider Microsoft Office. The overwhelming majority of business computing setups run it, so it’s a giant cash cow and a pillar of the company. However, they can only go on milking it for so long as they persuade consumers to buy upgrades. Stop and think about that for a second. Every product they introduce is a threat to their future cash flow. The better Office 2k7 is, the harder it will be to sell Office 2k9. The worse Office 2k7 is, the less likely it is that anyone will bother to upgrade from their perfectly adequate Office 2k5. In the office software space, over a long enough time period, Microsoft is totally, irredeemably fucked unless they can come up with a revolutionary innovation. Historically that has not been their strong point, and they are worried.
In that light, consider the console industry. It’s fiercely competitive, so they can play to their strengths and improve rather than innovate. Better still, its primary users are gamers, people famous for their willingness to spend a lot on upgrades and buy all the new toys as they become available. Best of all, gamers pay subscription fees. The other way out of Microsoft’s upgrade dilemma is for them to persuade consumers to buy software subscriptions instead of software lisences. They’ve wanted this for years and years, but every gesture they’ve ever made in that direction has crashed and burned…except for X-Box Live.
So why didn’t the X-Box pay off? No I’ve talked to really knows. My theory is that they were hoping, over several console generations, to capture most or all of the console market even if few users switch over directly. This is what they did with Excel: one minute Lotus 1-2-3 owned the market, then everyone upgraded their hardware and switched to Excel, and they became the new spreadsheet kings overnight. The thing is, they were able to do that because they could make the barrier to entry extremely low: Excel could do everything Lotus could, could read and write Lotus spreadsheets, the works. The cost for a Lotus user to switch to Excel was practically zero, so as long as Excel offered any improvements at all (and it did), they could easily take over. 90% of the time it didn’t matter which you were using, and the other 10% of the time Excel was better. But consoles don’t work that way; if you’re a guy who wants to play the Final Fantasy XII, you have to get a Playstation 2. There’s no way the X-Box can satisfy you. I don’t think they planned for that kind of brand loyalty, and their usual weapons are totally useless against it. But that’s only a guess.