The launch is done and you can’t go back in time and redo it. The failure has happened and now their best estimate is that they can make it operational by the end of November.
I haven’t heard anyone in the industry NOT call this a failure except you. Seriously, you are the only person. Less than 30 percent of people attempting can complete a transaction and you don’t call that a failure?
All of these have articles talking about healthcare.gov’s “failure”:
InformationWeek
Computerworld
Forbes
Healthcare IT News
CIO
How about this: any system that can’t complete 50% of it’s transactions is a “failure” (my bar for failure is actually much higher, but I’m humoring you).
Briefly, it’s identifying the real problems here, that have little or nothing to do with this website launch. The healthcare is industry is plaqued with incompatible and nonstandard standards. It’s a particular problem because of the history of healthcare software, which were some of the earliest large scale commercial products in the IT industry, and the isolated and competing interests of different medical institutions who had dreams of dominating the market. Modern healthcare software has to deal with a dizzying array of interchange standards which are constantly being updated with new and incompatible versions. When you move up from the protocol level to behavioral aspects of interchange software the problem becomes more complex. But anyone who is pointing at a particular standard or interchange mode as right or wrong is just offering an opinion. The great failure here was having the government take on this responsibility instead of just mandating that the existing healthcare companies deal with this themselves. A consortium of health insurance companies could have produced a much better product for much less money.
So at any time that your companies product did not operate and 100% of it’s transactions weren’t completed then the product was a failure. How do you guys stay in business on that basis?
It’s a huge problem to have wrong data in an EDI doc like that, but it’s also one of the easier things to fix.
Normally you test the shit out of that type of stuff because once you turn on the faucet, if there are any problems, it’s overwhelming to try to fix it on the fly and continue operations.
This is the second time you’ve followed this pattern. I post one thing, you respond by taking the opposite extreme, as if I had posted that.
Have I said anywhere that things must work 100%? Nope. Software is tough, there are always some problems.
Let’s look at what I did post:
“How about this: any system that can’t complete 50% of it’s transactions is a “failure” (my bar for failure is actually much higher, but I’m humoring you).”
As you can see I’ve asked you a legit question about a system that is less than 50% operational, but you responded as if I had said our systems are 100% operational.
And one of the biggest failures of this launch is a complete lack of sufficient testing. Any PM’s associated with this trainwreck should turn in their PMP certs and learn a new trade.
I’m using your own specs. Unless your products have never failed then for some period of time at least 50% of transactions have not completed. 100% is greater than 50% in case you didn’t know. Any time a computer was down, or your system didn’t operate it was failing to complete 100% of transactions. Now if you had provided a complete standard that specified the minimum time period to apply then I wouldn’t be able to respond like that, but you didn’t, you provided the kind of vague specification that allows problems to manifest. And now when it comes to your own products software is suddenly tough and there always some problems, but you won’t apply that same standard to the ACA rollout.
What are you talking about? I can barely decipher it.
If I had a product that went live on October 1 and by October 25 the rate of success for new transactions was still less than 30%, I would consider my project a massive failure.
As I said, the bar for my projects (and pretty much everyone I know in the industry) is much higher than that. If the first week of one of my projects resulted in 90% of transactions succeeding, I would consider that failure.
I’ll agree with that last part, I’m not sure how the law would be structured but a consortium of health insurance companies would almost certainly have rolled out a better product. Not necessarily because they have amazing IT people (they might–but they’d probably end up using some of the same contractors too that were actually on this project), but just because removed from the insanity of the government procurement process I think you could have had a much smaller number of contractors working on this in a much more integrated large team with someone at the top actually responsible ultimately for the whole thing. It’s not even apparent that we had that at all with this project, we know that CMS was handling pulling it all together but I don’t know if they had a manager who was ultimately over the project day-to-day or a committee that periodically reviewed stuff or what.
As to compatibility with health insurers, I don’t know that we’re talking strictly about that kind of error. I’ve heard about duplicate enrollments being sent over, for example. That’s not a problem with interoperability between different systems but some functional error in how the government’s process is creating and transmitting enrollment records.
Further, I don’t know if it was done this way but interop with the health insurance systems should be of zero concern. If I was in charge of this process I’d have said “we need to spec out a standard format for enrollment data, we transmit this to the health companies very early on, and it is their responsibility to make their systems accept the enrollment data in the standardized format we send it in.” I wouldn’t be doing special types of enrollment records for each insurer–and I hope the government isn’t doing that.
This is a situation where (in theory) several million customers are coming through a government system to a private health insurer, the onus should be on the insurer to make sure their systems can accept a standardized enrollment record, not on the government to customize for each insurer. The government does have the heavier club in that relationship and it’s best practice anyway.
TriPolar, I’m curious about your standards for when you consider your projects a failure or not. You and I clearly have different standards, which would be fine if we were talking a percent here or there, but in this case we are talking about much more.
Questions:
1 - Given that <30% success over 3 weeks is not considered a failed launch in your opinion, what percent of successful transactions would you consider failure?
2 - Is it your opinion that the launch or go live of a system never gets categorized as success or failure? If the system eventually gets working, regardless of chaos created or time and money wasted, you only measure the “launch” at the point in time where it either works ok or is scrapped?
3 - I know you said you have 40 years experience with large projects but it just occurred to me you didn’t say in what capacity. Maybe you’re a programmer, I really don’t know, but I do know your point of view doesn’t sound like someone that manages projects. Are you running the projects and you have the authority to delay going live?
4 - If you do manage projects, do you think 2 weeks of testing is reasonable for a large project like this? Only my smallest projects can get by on 2 weeks of testing (most recent just went live, was a 2 man month project, 2 weeks testing), so this one just doesn’t make any sense to me.
5 - When approaching the date to go live and adequate testing has not been performed - do you make the decision on your projects to go live anyway? If so, do you not have any negative consequence in your job when things don’t work? My projects, typically, would cost the company significant amounts of money if we went live and it wasn’t almost perfect. The amount of time and money to correct failed transactions is high and has ripple effects, so I don’t understand what your environment must be like that it’s not a problem when things don’t work right away.
Perhaps this science is not as settled as we think it is? I am willing to assume that none of you experts are bullshitting, that you all have experience and credential input/output the wazoo. Yet we have disagreements of the kind you don’t usually get in settled science.
Suppose that a group of computer wizards gets together and says “We can build this for about one hundred million, it’ll work like a charm and can be scaled up if the US population grown to one billion next week and can also double as a server for online games!”
And another equally credentialed and experienced guys look at it and say “Booooo-gus! It’ll cost at least $800 million and will crash on every login!”
How are we supposed to know who’s right, assuming that we don’t have the slightest idea what the fuck they are talking about? And if we can’t know, how can we expect our elected representatives to know? Perhaps the real source of our problem is that the science of massive database/interface is not as settled as we pretend or believe?
Are you seriously implying that the folks who run this sort of thing can’t be held responsible for fuck-ups? That just because you don’t know how to do it right, it’s OK if whoever is in charge doesn’t know either? Were you this generous with Dick and George when they underestimated the cost of the Iraq war? There have been many more web sites launched in the US than wars fought by the US. I’m guessing by at least a factor of 1,000.
I’m not implying shit, John. I’m asking a question entirely outside of my area of competence, i.e., snark and sarcasm. I do not pretend to know the answer. Which is why I am asking the question. Wherein liberal hypocrisy becomes pertinent I haven’t the foggiest, nor do I have the slightest idea why you thought it was.
I guess I just assumed that it was a given that if someone is appointed to head something in the government, that they should know how to make that something work. Is that not your assumption?
I don’t know jack shit about running an education department, but I sure as hell expect the Sec of Education to be an expert-- or at least know how to hire someone who is. Kinda goes without saying, no…?
Now, if it’s sending a man to the moon (something that has never been done before), I might cut the guy some slack. But, in fact, we did that OK. Launching a web site is no Moon Project. It’s been done a gazillion times.
My standards are quite high, but again, I consider failure to be something irreparable. I learned early in life to stay away from projects with ambiguous goals. In this case I don’t even know who failed. It’s possible the contractors actually did everything required of them by contract and the fault for any problems is entirely the government’s for not creating proper specifications. It could also be simply the fault of government for not managing the contract properly to make sure the requirements would be achieved. Either way the onus is on the government, and any classification of success or failure is not a technical one but something politically oriented. If this system is fixed by the end of the year it’s a lot more successful than the average government contracted effort.
There’s always someone to offer an opinion of success or failure, but to me there’s a way to specify that. If the contract specified that certain functionality should have been working to a reasonably precise standard by some date and that is not achieved I might call it a failure, but I’d tend not to waste my time on nitpicking the details if it was fixable long term. I also disagree with your categorization of chaos, I haven’t seen any rioting in the streets. On the other hand the waste of money bugs me a lot in this situation.
I call myself a simple country programmer, but there were limits to how much money I could have made that way. I’ve worked as a manager in companies, and then ran my own company for 17 years. I managed large scale projects taking months and years to complete, often in charge of employees at client companies as well as my partner’s employees and our contractors. As a project manager my job was to get projects completed on time and on budget meeting the exacting contract specifications that I insisted on. I have to admit I’ve never had to delay going live for my own projects, because I didn’t let that situation occur. I had more problems getting my clients to stop delaying go live dates because they constantly wanted to add more into a project. But I have operated as a consultant to companies and recommended holding up go live dates because they weren’t ready.
Absolutely not. I doubt there was any serious testing at all. Where do you get the 2 week time from anyway? If it’s just some media report I’m ignoring it. The testing process has to be specified in the contract. So can you tell me what testing was specified in the contract, or if the contractor failed to do that?
If you want to know how important this was to me, I’ll tell you that I’ve had to wait up to two years to get paid for completion of contracts, knowingly, because I made it clear up front that it would take the clients longer to verify the functionality than it would take to do the work. You think software is tough? Try getting end users to properly test the product the you’ve provided. For end user applications testing requires real data and users. I much preferred working on systems software where testing can be done with automated tools.
No, I’d never recommend going live with a faulty system. But faulty has to be pre-defined. And I’ve often stepped in to fix the problems created by others, and often while people were crying that the sky was falling it took very little time or effort to fix the problems. The biggest problem in those cases is the pessimistic attitude instead of dispassionate evaluation of what’s actually wrong and what it takes to fix it.
I know what failed software is. Failures of others provided a lot money for me. My company was often called in after actual failures, where a contractor was paid millions, or millions were spent inhouse without actually producing anything, or so little that it was laughable. That’s what a failure is. Good managers don’t apply that word to minor milestones, it’s not a good way to manage, it represents a “can’t do” attitude, and has nothing to do with resolving problems and salvaging projects.
The consortium could have been formed simply for the purpose of creating the standard, then each member, and anyone else could use that standard to produce a system. There doesn’t have to be just one portal to health insurance exchanges. I still don’t understand why the government should be involved in the development of this software, or maintaining websites.
Actually, this is not a bad question at all and is exactly the source of many problems.
First thing to do is to understand from the start that large projects like this can only be successfully managed by people with similar experience. For example, let’s say the largest project I ever managed had about $10 million in labor costs, I would almost for sure fail to deliver on time and on budget a project with $100 million in labor costs. It’s tough to just make that leap, you need to work your way into larger projects.
So, if the person in charge of this $600 million project has never managed a similar sized project successfully, maybe you couldn’t find anyone because that’s pretty big, then you better put something in place to lessen the risk. Maybe a small core team of managers (3?) that can work together. Not sure if that would work, but you better not just put someone in charge that happens to have a pulse.
Secondly, let’s look at your $100 million success vs $800 million failure scenario:
While there will be disagreements, people with a good track record in their field will typically not differ by that large amount. That wild variation implies someone is estimating without experience or outside of their area of expertise. As a person in charge you need to look for a background in similar projects and size.
Third, when you are into new territory, you break it down and work on pieces, build prototypes/proof of concept, scale it, etc. For example, I was working on a project that required some new style of transaction processing that our current system didn’t handle and it needed to be much higher volume than anything we did previously. Part of the project was to build a test version of the basic underlying infrastructure and see if it scaled to the level we needed it. This happened before we committed to building on top of that design. Once we validated it would work,we continued the design process that integrated with that system and when we went live we knew exactly the characteristics of that system and it has scaled without problem.