First, it’s probably a misconception that people were logging in on Oct 1 to buy policies for Jan 1 - people were logging in Oct 1 to see what they could buy and plan.
Second, health insurance and insurance in general usually requires pre-planning and long lead times (particularly when individual health insurance had health prequalification). Frankly I’d rather know what health insurance I am considering before oct 1 for January 1st, let alone waiting until November to even see it.
I do not buy the claim that demand outstripped their estimates. First of all, the estimates of how many hits they planned for sounds completely unrealistic. Second, even the higher number of hits should be easily manageable in a modern scalable architecture design.
My understanding of the real problem here is that the entire front end architecture is faulty. It pushes way too much data to the client, and it requires far too much handshaking back and forth during a session.
From Reuters:
Other web sites have been linking samples of the HTML code inside the pages, and it sounds like utter garbage, plus a lot of ‘To Do’ placeholders. Couple that with typos and grammatical errors on the web site, and it sure sounds to me like this is a case of a a project that got out of control and then pushed out to the public to meet political goals instead of waiting for proper load testing and QA.
If the analyst at Reuters is correct, this is pathetic. Anyone who has written scalable systems can spot a number of very basic errors. For example, you don’t upload all the code for all browsers every time, relying on templating and Javascript to configure the site for the browser. Instead, you look at the browser ID in the header and send the correct page without all the extra junk in it.
Dynamic Javascript with server calls in the background can be wonderful when making sophisticated web sites, but when you have to scale to millions of consecutive users you have to keep that crap to a minimum. When a single request from a client results in 90+ round trips to the server and an exchange of megabytes of data, your system isn’t going to scale. Sorry. There’s a reason why Amazon, eBay, Paypal and other high-volume web sites use simple designs with minimal round trips to the server and low graphical content.
I’ve tried to sign up several times. It always gave me the message that the server is unavailable at this time. Until today when it sent me a confirmation email, despite the fact that it had already told me that the server was unavailable. Unfortunately, apparently the email has a time limit (now there was a wonderful idea to make it harder on us) and I didn’t see it and click on the verification link in time. So, do over.
They’ve also got it set up so Firefox doesn’t ‘remember’ the stuff I’ve already typed into the fields so I have to type it in all over again ever single time I start over.
I hope they get this fixed before the penalties kick in but I doubt that’ll happen.
not aimed at you. Sorry. I’m just generally amazed at the level of competence of the people we rely on to spend our money wisely.
What pisses me off is that I expected a clusterfuck AFTER people enrolled… We can’t even get out of the starting gate. They certainly need to roll it back enough to compensate for the delay. Believe it or not, some people have to work for a living and those on the lowest rung have multiple jobs. They don’t have time to be the beta testers for this software. I’ve been to websites that estimate what the costs will be but they didn’t have any details. Not a government website so I can’t bitch about it but the price quoted made no sense compared to known insurance quotes and I wanted to do a comparison to see what’s what.
Even though it should be obvious, it never occurred to me there’s a market for clouds. Interesting reading…
The Cloud that broke (better verb needed?) yesterday: Verizon “Terremark.” Specifically, its
Enterprise Cloud: Federal Edition
Combines the power and flexibility of infrastructure-as-a-service with the expertise, security and availability that government agencies with mission-critical computing needs demand of their infrastructure.
Government Cloud Computing
This federal-only cloud infrastructure (Community Cloud, as defined by NIST Version 15, 10-7-09) is hosted at Verizon Terremark’s NAP of the Capital Region in Culpeper, Va. and the NAP of the Americas in Miami, Fla., Tier III data centers that are purpose-built to house federal IT infrastructure.
An easy-to-use Web-based management interface provides command-and-control over a cloud-based resource pool of compute, storage and network built on a fully clustered enterprise-class computing architecture featuring virtualization technology from VMware, and compute and network infrastructure from Cisco and NetApp.
All I can offer is my own experience. I never could figure out how to set up a password. I designed a password that fit the requirements, but it kept saying I didn’t use the correct characters. If this was caused by high traffic, then that would mean that, if the message fails to go through, it defaults to saying something is wrong with the password. That’s ridiculous.
The other idea is just that the instructions did not match the requirements–which, while less ridiculous, is still pretty bad.
All of this would be design, would it not?
Amazon and Google evolved and slowly developed over decades. And have had many growing pains during the process.
This site was created from whole cloth in, what, a year and a half? And that’s being generous and assuming the requirements weren’t changing during the process, which we know they were.
It’s apples-to-oranges.
Well, I’ve always used Yahoo Messenger for my IMing needs — much-lauded private enterprise, and a big enough web giant to rival Google or Amazon.
A few months ago I tried to log on, after not being on for a while, and was told to change my password. I entered my old password, and chose a new p/w with KPassGen maybe 20 characters: told only alphanumerical characters on rejection; made a dozen passwords ranging from simple to overboard before abandoning my old email/IM account forever, No password was acceptable, including the old one, for I then found it was a known problem with Yahoo! that new passwords were unaccepted. And they still demand that passwords be changed after a period of disuse.
Insane decisions are not limited to government.
I don’t recall Amazon and Google crashing out of the gate. In fact, their success was because they worked and worked well.
This is pure incompetence thrust upon the public who are now expected to beta test it.
First of all, if you don’t remember the major Amazon crashes then that’s just an artifact of your memory. (Admittedly Google has always been pretty steady.)
Secondly, my point was that Google and Amazon built up their levels of traffic over a long time. They didn’t have 50,000 simultaneous users on their first day, they had years to get their servers and applications into shape to handle the traffic. That’s the critical difference.
ACA was created in 2010. They had years to get their servers and applications into shape to handle the traffic.
They adapted the Canadian version, which was pretty simple:
Are you Canadian?
Yes - Good, you’re covered.
No - Sorry, can’t help you.
If you demonstrate that the site was completely specced-out in 2010, I’ll concede that point.
I’m 99.9% sure it was not until much, much, later. And even after the original specifications were published, they were edited late in the development process.
Amazon did not choose the roll out date. That was done by politicians. As has been discussed, the designers were continually barraged with changes. It was never tested. Which means it never had a director who knew what he or she was doing in the first place. Beta testing was left to the enrollee. It was a political version of the Homer Simpson car.
Right.
And to be fair, I’m not saying the software company that created it is super-competent, best-in-the-universe-blow-you-away. I’m saying that nobody could have done a better job given the same constraints.
Maybe it’s slightly more of a mess than it would have been if, say, Amazon had designed the site. But I doubt it. And Amazon doesn’t do that kind of work anyway, so it’s moot.
I think Sebelius is probably going to have to fall on her sword over this.
I don’t see where you can blame the software company when the government who ordered it continually made changes. This was managed by the government and that goes all the way to the top. It’s the job of the President to oversee these projects just like any other CEO. He has to ask for updates and be shown that it works and is on schedule. None of this appears to have been done. This isn’t “sorta” problematic. It’s a complete hose up.
What makes this worse is that the President ran on a platform of transparency and he has gone out of his way to prevent public disclosure of how bad it is. He’s got the time to parade the few who were able to complete the process but has forbidden disclosure of actual numbers. He also stated people would not have to give up their insurance and that it would lower rates. Both of these statements are not only false but he would have to know they were not and are not true. It is impossible for rates of established policies to go down when additional levels of insurance are raised.
Not it isn’t. In fact, if young, healthy people are required to buy insurance, is is impossible for rates to not go down.
This is a management problem that flows from the President on down. This is the result of having a chief executive with no executive experience.
Here’s the way this would have been done if there was a competent chain of management in place:
[ul]
[li]The whole web portal part of the law would have been subject to a feasibility analysis to make sure it could even be done within the timeframe specified. That includes accounting for the time it takes for the law to be turned into a set of detailed regulations, and for those in turn to be turned into detailed requirements for engineering. [/li]
Requirements analysis and planning for a project of this scope could have easily taken six months to a year after the regulations were set in stone, and I believe those were still being drafted as of last year.
[li]A risk mitigation plan should have been put in place, recognizing that the portal was a critical portion of the overall law. That mitigation could have included cutting features, standing up a manual temporary process, or other ways to make sure the whole thing wouldn’t come collapsing down if the deadline was missed. [/li]
[li]The project plan should have been developed, and signed off by all stakeholders from the President on down to the leads of the development teams. That plan should have identified the critical paths, called out for tollgate reviews at critical points in the process, and clearly defined the conditions upon which potential problems escalated up the chain of management. That plan would also include the trigger points for risk mitigation (“If we can’t hit milestone 3 by date X, we will revert to the alternate, simpler plan”).[/li]
[li]The President should then have appointed someone to be at the top of this project with responsibility for reporting problems to him if the escalation plan reached that point. For example, if the schedule slipped by more than a month, or a risk was uncovered that threatened a delay of more than X, or if the budget was running behind by X amount. By making it specific, you avoid the problem of an underling deciding not to report a problem because can ‘handle it’.[/li]
[li]The President should be having regular meetings with various staff members, including the person in charge of that project. That person should be giving regular status reports as called out in the project plan so that the President understands whether they are on track. He doesn’t have to micro-manage anything or know a thing about software development - he just has to satisfy himself that things are going according to the plan and that the people he trusts to oversee it are satisfied.[/li]
[li]If the president was unsure of his knowledge or capability to determine that the project plan was realistic and contained the required elements, he should have brought in an outside review board or experts to go over it and advise him as to whether the plan was realistic and controllable, and whether or not it missed key elements like security analysis, penetration testing, integration tests, and whether enough time was allocated for those activities.[/li][/ul]
Those of you who think CEOs don’t matter and don’t earn their pay could learn something from this. In my experience, bad management flows down from the top. I’ve been in one organization long enough to see what happens when you have a bad CEO and a good one. Bad CEOs are the ones who berate their subordinates for bringing them bad news, or respond to requests for help with answers like “Just get it done”, or who constantly assign blame or deflect responsibility to the layer below them.
The CEO is the ultimate ‘go-to’ guy: The one who, when all else fails, can break through the organization logjams and right the ship that’s heading for the rocks. When you have a good one, everything starts to line up, everyone knows what they should be doing, clear lines of communication are established, and infighting between groups is at a minimum. When you have a bad one, everyone starts looking for cover or protecting their turf, people become cynical and respond to requests indifferently, information stops flowing as people become afraid to communicate or feel like they’d be wasting their time to do so, etc.
This mess has all the earmarks of management troubles from the top down. No one knew who was really in charge, when critical problems surfaced they were buried somewhere in the bureaucracy, etc.
When Obama said “No one is madder than me!”, that’s a big clue that his organization is completely dysfunctional and that he doesn’t understand the job of the Chief Executive. He wasn’t the victim here - he was the perpetrator. What he should have said was, “This was my fault. I should have been on top of this. I will get it fixed.”