That article is pretty interesting. The fact that Sebelius was excited to turn on the system (“we are about to make history”) shows how completely clueless they were.
A complex system that hasn’t been tested and they were blindly expecting success. That is just crazy.
And after reading more about how they ignored good advice up front, I have absolutely no sympathy for Obama or his team.
Did I miss something in the article? I bet they were neither, because a) they did the vast majority of their debugging and verification against staging environments and b) they did take some significant downtime Friday evening.
I agree, they wouldn’t have been working in a production environment, they would have been in some sort of dev/test environments. And the logic of leaving the site up to expose all of the problems makes sense even though it’s a painful way to handle the situation for the end users.
Obviously some code has to be fixed in dev/test environments, but it’s not clear how much. Because if they have a completely parallel dev/test system, it makes no sense to say you’re making ‘live’ fixes without taking the system down. Ideally you’d just leave the system alone and do all the fixes on the test system, and create a release cycle for migrating new code from the dev system to the live system.
I suppose they could be doing hot fixes for the most serious bugs, taking the system down for short periods to install new code, while planning larger bug fix releases on a longer cycle. Thinking about it, that must be what they’ve been doing - the Nov 30 date was probably an internal release data for a dot-level update, and in the meantime they’ve been hotfixing critical bugs as they found them.
One of the complexities of building a system that must connect to large legacy systems is that it can be very hard to test that without those systems in the mix. If you have tightly controlled interfaces you can simulate them, but that makes the assumption that the simulation has captured the real-world behavior of those systems - which you won’t know until you test them live.
Anyway… here we are at Nov 30, and the system is still not working properly. Not even close. Apparently, they’ve abandoned calculating individual payments to insurance companies and are doing some kind of offline bulk payment scheme based on self-reporting of income. So it sounds like they’ve ‘fixed’ the system by developing questionable workarounds for the bad code in the back end instead of getting it fixed.
CNN tried to use the system today to file a plan, and it crashed on them.
I’m sure the site is better than it was last month - how could it not be? Whether it actually substantially ‘works’… we’ll find out in the next few weeks.
System had 25% error rate for data sent to insurers
Looks like it’s manual cleanup time