A detailed report from the Washington Post on Saturday, titled HealthCare.gov contractor had high confidence but low success, revealed more information about some of the troubles behind the scenes during the development of the healthcare.gov website. It’s a lengthy read, but it’s recommended you read the full article to get the full scope of the story.
The report provides further background on the work performed by the project’s primary contractor, CGI, and the growing concern from federal officials that the work was not proceeding as expected in the months and weeks leading up to the launch.
The article begins:
At 9 a.m. on Aug. 22, a team of federal health officials sat down in a Baltimore conference room with at least a dozen employees of CGI Federal, the company with the main contract to build the online federal health insurance marketplace. For six weeks, the federal officials overseeing the project had become increasingly worried that CGI was missing deadlines, understaffing the work and overstating its progress.
As the meeting began, one of the officials reminded the CGI employees that HealthCare.gov was “the president’s number one priority,” assured them that the discussion would be a “blame-free zone,” and then bored in. “We must be honest and open with each other,” the official said, according to documents obtained from participants in the session. “I have to know what I don’t know.”
The top CGI executive in the room sounded contrite. “We recognize we have to build trust back . . . ” said Cheryl Campbell, the company’s senior vice president in charge of the project.
For that day and the next, CGI staff huddled with government officials in the semicircular conference room at the headquarters of the federal Centers for Medicare and Medicaid Services (CMS), the agency overseeing the project. They combed through 15 pages of spreadsheets they had brought, which spelled out the company’s level of confidence — high, medium or low — that individual components would be ready.
The article goes on to cite a final “pre-flight checklist” a week before the site’s launch that showed “41 of 91 separate functions” that were still not working, as well as a spreadsheet dated the day of the launch identifying “30 defects on features scheduled to have been working already,” including 5 classified as “critical.”
A government official described that while the contractor often met deliverable deadlines on components during those weeks before the launch, “they contained such faulty computer code that features did not hold up under closer scrutiny — or failed later if more than several thousand people at a time tried to use them.”
The article also cites details from some of the emails that were recently released by House committees investigating the rollout issues, in which officials on the project share concern about the contractor’s performance.
As the launch drew closer, officials tried to mitigate issues by scaling back some of the scope of the remaining items, as is detailed near the conclusion of the Post’s article.
By the time Campbell and CGI employees drove from their Herndon offices to CMS’s Baltimore headquarters, the agency’s staff had decided it had no choice but to try to find out exactly how far along the contractor was, and set priorities for having less done by the launch date than anyone had hoped. Toward the start of the meeting, one official asked the CGI team point-blank: “How much of the function is developed? How much is left to develop? How much of the function’s requirements will be met? How sound is the development?” according to documents from the session.
CGI’s rankings of high, medium and low were only for parts of the insurance exchange that, at that point, were supposed to be ready by Oct. 1. The two spreadsheets they presented show that, for each item, CGI employees had written a date when they said it would be built and tested enough by the company that it was ready to be evaluated by outsiders. During the meeting, CGI employees acknowledged that they considered an item ready for outside testing if the computer code for it was 75 percent ready, documents show.
Of 95 elements that CGI evaluated, employees said they had high confidence in 45 of them. Of the 45, 16 of the high ratings were for SHOP, the part of the exchange to sell health plans to small businesses — a function where further testing found that it worked so poorly that it is still not yet available.
The “anonymous shopping” feature — in which consumers were to be able to browse health plans without creating an account — also was rated as a high-confidence item. It never made it onto the “pre-flight” assessment because, by then, government officials had decided that it was too flawed to offer at the outset.
A senior administration official, who asked not to be identified because of the sensitivity of the matter, said he was not aware of any indication the “pre-flight” review ever reached the White House.
Of course, there are numerous contributing factors for the problems with the healthcare.gov launch, which include the lack of a single empowered decision-making authority and the complexity and flaws with the law in general, among others. But this article offers some insight into some of the development and project management challenges.
It’s a lengthy article with a lot of information. While some of the details have been previously reported, it’s presented all together in the context of the weeks leading into the launch. Read the whole thing at the Washington Post.
DONATE
Donations tax deductible
to the full extent allowed by law.
Comments
“Of 95 elements that CGI evaluated, employees said they had high confidence in 45 of them. Of the 45, 16 of the high ratings were for SHOP, the part of the exchange to sell health plans to small businesses — a function where further testing found that it worked so poorly that it is still not yet available.”
I can’t decide whether to get mad, or just sit down and have a good, long cry over just how criminally inept, stupid, and dishonest this administration is.
They knew the website was an unmitigated disaster and they had the gall to have their big rollout and then all stand around with surprised looks on their faces as if they had no idea it would be a complete failure.
IF no one told Obama, I imagine he’d be so pissed off he’d be firing people left and right. But he’s not, is he?
Just think – what if, instead of blowing 650 million, they had called a meeting with Jeff Bezos (Amazon) and Larry Page (Google) and said ‘Guys we have this little project for you and your staff. Seeing as you already have the staff and infrastructure in place, with a decade or more of this kind of massively high-traffic, commercially sensitive kind of thing, we want to hire you to build this for us.’.
What’s that you say, you need $ 250,000,000 for each company, and it will take 4 years ? OK, where do we sign ?’.
Imagine if HG.gov worked as well as Google or Amazon, had that kind of connectivity, bandwidth, storage, and compute power behind it, and the support structures of Amazon and Google.
Naaaa…..
Though the whole project was FUBAR from the start, your analogy misses the difficult technical challenges. Amazon or Google don’t have to interface with a bunch or legacy COBOL batch systems that were never designed for interactive remote access. I suspect, but don’t know that many of the time-out issues are the result of those database fetches against legacy apps.
IMHO they should have left the legacy apps alone and built a datawarehouse in the CMS facility that would be updated nightly from the systems of record. Then the Obamacare web site would have much faster and consistent response…
I would bet that a LOT of the government systems are legacy COBOL. And you can forget about ‘database fetches’ – there WERE no ‘databses’ back then. It was all flat file. The contortions we went through to extract datasets were a trip. Offline batch runs extracted keys and relative record positions (what we call Primary keys and secondary indexes today), then sorted them. To ‘fetch’, you had to tree-search the little ‘guide file’ you created, then direct-call the large source. Kind of like a ‘many to many joining table’ today, but only 2 columns instead of three. Today you would put a lot of that in DB, doing nightly ETL’s. So you’re either going to find they already have what you need and you can borrow on it, or you have to write (or get them to write) a process to add to their runs on their end. Even if they have some kind of add-on ODBC connector or such, you can count on it being slower than all hell.
To think that you’re going to do that with a BUTTLOAD of legacy systems (each agency is going to have 20 or whatever that you need to figure out) from a BUTTLOAD of agencies, and wrap it all up with a bow on top all in one shot is insanity. and then roll it out to 100,000,000 users all in one shot ? Geez…. when it failed load testing for 500 ?
I read that they are actually doing the whole thing in MYSQL. Which apparently no one there knows. Nice.
MarkLogic NOSQL
same idea. some tool, selected by the customer rather than the tech team…
The problem with PPACA isn’t the website. It is the pricing. It’s guaranteed to drive people away from coverage.
If Democrats and the media ever really thought mandating a laundry list of normally optional and sometimes costly coverages as part of every single plan was going to “bend the cost curve down” and “save the average family $2500 per year,” they should never leave the house without their helmets on securely. Too much brain damage already.
But…but …. if your policy goes up $ 2,000, but you get $ 3,000 a year worth of free birth control pills, like that crazy libber bitch they had on TV saying that a few years ago, just think ! oops… what did you say ? $ 9 a month at Target for the exact same brand of pills ?
Well, but …. at least I’m covered for pre-natal care, in case this 57 year old single guy ever gets preggers !
Didn’t Michelle Obama and Cheryl Campbell carpet munch their way through Princeton together.
Have no fear, half of the 650 million development cost of healthcare.gov will find its way into the DNC’s coffers or those of OFA. That was the reason for the no bid status, of course. This is the first presidential administration that could likely be prosecuted under the RICO statutes.
The USPS just gave out a ‘no bid’ contract to sell 56 Post Offices as ‘extra inventory’. The value is in the billions.
What real estate agent got that NO BID contract ? And thus the commissions on all that ?
Diane Feinstein’s husband.
No shit.
Interesting article I found linked to by Instapundit: Healthcare.gov and the Gulf Between Planning and Reality, esp for those who have been in the IT trenches, as some here clearly have.
The time to get really, really scared is when a bunch of liberal arts grads, many who later went to law school, intoning that “Failure is not an option”. Pretty much guaranteeing that not only is it an option, it is now highly unlikely.
Correcting your typo (failure is …) highly LIKELY.