Knowing what I know about the government, I wouldn’t be surprised if they included many times more code than was necessary. But as the “500 million” figure comes from a single article in the New York Times, it may be wrong.
Not “may be wrong”, it’s just plain wrong and it was clearly written by someone that has absolutely no clue whatsoever.
It’s the equivalent of saying they just built a building that is 10,000 feet tall.
Weird thing is that a lot of experts, familiar with the same facts as you, have said that the 500M lines is a lot, and a sign of an inefficient process, but have not reached the same conclusion that it’s just plain wrong.
The only people makig that assertion that I’ve seen are in this thread.
1 - Anyone that thinks 500m LOC for that project is in the realm of possibility has no experience with large software projects.
2 - Let’s do a little math, but a little background first. Lines of code per year (productivity) drops as project size increases. A project that is 10 million LOC has productivity of about 1,600 LOC per YEAR on average.
Now the math:
500,000,000/1,600=312,500 man years for that project
And that’s being generous with the estimate.
I agree. I suspect that someone gave the NY Times an inflated figure - possibly by counting all the lines of code in every open source library they are using, plus perhaps lines of code in the existing legacy systems that are part of the new system. Or perhaps they are counting machine-generated code or something. Or perhaps they wrote a million lines of code and deployed it on 500 machines and they’re counting that way. Which would be stupid, but then everything about this project seems to be stupid.
I can’t believe that 500 million lines of new code were written by developers in the last year or two.
I hope the people who keep clamoring for more government regulation of private industry and more government involvement in the economy are noticing just how lackadaisical government is about quality when they’re the ones building things. The Washington Post reported that end-to-end integration testing didn’t even start until Sept. 26. If so, that’s not just bad management - it’s a straight-up negligence and a violation of proper engineering practices so bad that if a private company did that and their product harmed someone they could be held liable for negligence.
And if a private company did anything remotely this bad with such an important system, Obama would be leading the charge for new regulations. But who watches the watchers? If government regulation of the marketplace is so important, what happens when the government IS the marketplace and totally above the law?
Perhaps they C&Pd a lot of it. I don’t know.
You could be right. What I’m saying is that it’s weird that the experts who have been interviewed by the media have not said that it’s completely inconceivable. If these arguments are as valid as you make them seem, one would think these various experts should realize them too.
Another perplexing aspect of this: The government hired 55 outside contracting firms to develop the project, but they didn’t hire an integration lead. That’s bizarre. Every large project like this is generally assigned an integration lead who is responsible for making sure all the pieces come together and work properly.
Instead, the government managed the integration itself through the Center for Medicare and Medicaid Services - an agency that has no experience whatsoever with large IT projects. That’s just crazy. There are only three possibilities I can think of for this grossly stupid act:
- They couldn’t find anyone willing to take on the project under the constraints given, knowing that it was likely to fail.
- Hubris. This administration thinks it’s so smart that it can do anything. How hard can a web site be, really?
- They wanted to retain political control over the process so they could roll out the web site on their schedule and not on a contractor’s schedule.
In any case, it was a horrible decision.
Another bizarre aspect of this: Kathleen Sebelius says that Obama knew nothing about the problems with the project, and first heard of them in the media like everyone else. If so, that’s gross dereliction of the office. Obamacare is his signature legislation. Was he really so disinterested in it that he never even bothered to check how it was going? Does his science and technology advisor not keep on top of such a major project? Did Sebelius not think to inform him that the project was at risk? Did SHE even know? This was, after all, the biggest project going on in her Department for the past 3 years.
The incompetence being displayed here makes “Heckuva Job Brownie”'s management of Katrina look brilliant.
This Washington Post article is incredibly damning.
First, the administration’s original spin that the site failed because of ‘unprecedented demand’ was a bald-faced lie. From the article:
So they knew before the launch that it couldn’t withstand even a few hundred users. But the really damning thing about this is that they only carried out their load test ‘a few days’ before the rollout date. That’s insane.
Just for comparison, my software team is about 30 people, and when we do a six-month project, the last two months are generally scheduled for load testing and integration testing. On bigger projects, it’s not unheard of for the testing phase to be six months, a year, or even more.
For example, Windows 7. which was only an incremental release of fixes from Vista, was released to select partners for testing in January of 2008. I guarantee you there were already months of testing in-house before then. The beta of Windows 7 was released a year later through Technet and MSDN, going out only to developers and integrators and such. A ‘release candidate’ with all the proposed final changes in it, was released in May of 2009. The public release finally came in Oct 2009, almost two years after the first alpha release. This is not unheard of for large complex systems that must interact with many external systems. The permutations are huge, and a lot of testing has to be done to make sure everything works.
I’ve never seen a project of any large scope that didn’t find some critical bugs during integration testing. You always have to leave enough time to fix those bugs. You can’t schedule your testing for a few days before release, because if you found something you’d never have time to fix it. So the people in charge of this project either absolutely knew they were foisting a broken product onto the public long before they did their final testing, or they were so incompetent they had no idea what the proper software engineering practices were.
Then there’s this:
So they knew of the problems long before the rollout. They didn’t bother end-to-end testing until five days before go-live. Their only, very late load test failed at a tiny fraction of the load they expected. And because of those failures, I’m sure they never even managed to test some of the back-end integration before they went live with it.
Then they rolled out the site anyway, it broke as it had to, and they blamed the failure on “an incredible amount of interest” and said that it showed how popular the program was. When they couldn’t get away with that any more, they described the problem as ‘a few glitches’. And of course, Obama doesn’t take the blame for any of it. He’s ‘madder than hell’. In other words, he’s a victim of the problems just like everyone else.
This is gross incompetence, and professional malpractice. Any engineer who signed off on this would have had his career ruined. And the amount of bald-faced mendacity coming out of the administration is breathtaking. They’ve lied about this from the beginning, and they’re still lying about it.
The health execs were right - there should have been a pilot project, then the final system should have had a phased release. It was released to the public at least a year too early. This was not done accidental, and none of these failures should have been a surprise to this administration. If they were, they’re incompetent. If they weren’t, they’re liars.
The only estimate I’ve read by someone in the software industry was 500,000 lines of code, which is very possible.
I very much doubt that President Obama wrote any of the software. Perhaps he slaps out a few hundred lines of code for relaxation or meditation, unwind from a hard day of trying to reason with fanatics. Sort of the Vulcan version of Zen. Nah, he probably thinks a “compiler” is support staff that gathers relevant files from the cabinet.
And you’re just as quick to excuse corporate CEOs for the excesses of their companies, right? Obama is the chief executive of the federal government. He is responsible for the actions of his underlings, just as you rightly held George Bush responsible for the Katrina response despite the fact that Bush is probably not an expert in disaster management or emergency relief. It still fell on Bush because he was the man ultimately in charge, just like the CEO of my company would be held accountable if my team made some egregious error.
But Obama has a habit of deflecting blame, doesn’t he? It’s amazing how many failures he first hears about on the news. Harry Truman must be spinning in his grave.
Are you going to specify any of these amazing failures? Or simply allude to them as if they were something to which we had already agreed upon? Playing a little fast and loose there, aren’t you, podnuh?
Listening to Sebilius today saying that “no is more upset than I am” made me laugh (and not “hah-hah” laugh). I suspect people trying to get into the exchanges who are unable to are “more upset” than she is. And to call the fix a “Tech Surge” made me go 
. Why would Obama name this after something he voted against (“The Surge” in Iraq) when he was a Senator?
Something else you should realize is that when the media interviews “experts”, you really have no idea whether they are an expert or not.
Here’s a perfect example:
“After assessing the website, Dave Kennedy, the CEO of information-security company Trusted Sec, estimates that about 20 percent of Healthcare.gov needs to be rewritten. With a whopping 500 million lines of code, according to a recent New York Times report, Kennedy believes fixing the site would probably take six months to a year.”
Dave Kennedy’s background is security. He doesn’t build large software systems and doesn’t have any background in them. So he may be an expert on security but not an expert in large software systems.
Here are comments from people that do have a clue:
"Yesterday I wrote about the New York Times article citing an anonymous “specialist” who made absurd claims about the software that drives the new Healthcare.gov website, particularly that the web application consists of 500 million lines of code (highly doubtful) of which 5 million need to be fixed.
Here’s a really good piece with more on these ridiculous statements, by David Auerbach: Healthcare.gov Problems: What 5 Million Lines of Code Really Means."
and another:
“And then, to make the media’s tech fail even more ridiculous, we have the New York Times making the absolutely absurd claim that the Healthcare.gov web application contains 500 million lines of code”
“The ACA website is a government project so there’s undoubtedly some middleware bloat involved, but half a billion lines of code is simply an absurd estimate. That’s either a typo, or the Times’ “specialist” is playing a prank on them.”
And another:
"Hi Josh, I have been writing software for 35 years and have to say that these quotes from the NYT article are very suspicious
“One specialist said that as many as five million lines of software code may need to be rewritten before the Web site runs properly.
"
“According to one specialist, the Web site contains about 500 million lines of software code.”
I assume it is the same specialist in both quotes, who is either lying or being lied to. 500 million lines would put healthcare.gov among the largest program in the history of humanity, if not at the very top.”
The real scandal here is the price tag. I don’t know what an accurate figure is for this application, but it sounds like it’s somewhere between 10 and 100 times too high for the functionality involved.
Reports today indicate that many of the problems are disappearing rapidly, typical for the roll-out of a product like this under these circumstances.
Here’s one possible way to get a sense. According to that same NYT article:
Question is if the second sentence is true. If it is, then maybe the bigger number is possible. From the scale of what you guys are saying, it sounds like the claim about large banks’ computer systems is also inconceivable.
Nothing would make the bigger number possible because it simply isn’t.
You just don’t realize the magnitude of ridiculousness of that number.
Contracts were signed in Dec 2011, reports are that they started coding this spring, but let’s pretend they had 2 full years with 300 working days per year, that’s 833,333 lines of code PER DAY.
833,333 lines of code is a good sized system in itself that would takes months just to fully test every function/action/permutation.
Regarding banks: they do have tons of code that has been built up over the last 4 or 5 decades. Same with SAP and their 250 million LOC, they’ve been building it since the 80’s.
Here is the diagram that Sam Stone posted. Could the 500 million number be a sum that includes all of the lines of code in all of the existing systems that Healthcare.gov interacts with?
I’ve already addressed this and you’ve not responded.
There’s no point in repeating how many lines of code it would be per day etc., because I’m suggesting that they may not have coded 500M unique lines from scratch.
-
There could be LOC which are duplicated multiple times throughout the program.
-
More significantly, the software outfits who were hired to create the program presumably have some experience creating programs that are at least somewhat similar to this. It’s likely that they already have a lot of code that performs a somewhat similar function to various components that are required for this program. So it’s possible that they took a bunch of pre-existing code and plunked it into this program with some customization.
Like 62% of all statistics, I assume it was made up on the spot.