If you have "no place to go," come here!

ObamaCare Clusterfuck: First hints of real back-end problems on the Federal Exchange begin to appear

Via O'Reilly -- the highly literate and excellent tech publishing company -- we read the following. Note that the grey-haired[1] tech guy has pronounced judgment in the headline:

What Developers Can Learn from
... Remember, even a failure can serve as an example of what not to do
2) Pretty doesn’t trump functional. The site is very well designed from a graphical perspective, and is clearly using lots of Javascript and AJAX to do snazzy transitions and requests in the background. Unfortunately, it doesn’t seem to be interacting with the intermittent failures on the backend very well. If you’re going to make requests behind the scenes, you need to be very tolerant of failures. The site seems to fail silently and leave a broken user experience in its wake, with no way to continue. Nothing drives a user crazy more than having to go through the same form over and over because of failures that leave them high and dry.

That's why I'm wondering if backend[2] failures are the cause of site outage experiences like Rainbow Girl's, or mine. I'm guessing yes. [Others agree. "As few as 1 in 100 applications on the federal exchange contains enough information to enroll the applicant in a plan, several insurance industry sources told CNBC on Friday. Some of the problems involve how the exchange's software collects and verifies an applicant's data." ] So let's look at intermittent failures. I should stress that although I've done work of this kind, I've never done it on this scale; to someone at the top of the systems building power curve, I'm an intelligent layman with good research and critical thinking skills. That said, let's try to dope out what's going on, given that the Obama administration has been extraordinarily secretive about this project, so we're trying to figure out what's going on inside a black box.

First, intermittent failures are the hardest to track down, since intermittent failures are just that: intermittent. (Here's a real life story where the guy basically twiddled stuff in his vehicle until the #FAIL went away, rather like an ant pushing a crumb randomly until it falls into a hole in the nest. And this post makes the point that intermittent failures can be, institutionally, a cancer.) Here's what a testing consultant to software firms has to say about the heuristic of finding intermittent failures. I'm going to highlight two important characteristics of their heuristics: Institutional issues and critical thinking skills . (There are many items that have to do with technical skills and observational skills, but in a multi-institutional project, as ObamaCare, I suggest those come under the heading of institutional skills. ObamaCare is not a corporation of systems with one owner, but a federation of systems with many owners, so getting to a place to perform observations and exercise technical skills isn't a given; it will take at least one meeting. But heck. They've got 88 days. Counting holidays.)

Some General Suggestions for Investigating Intermittent Problems:

  • Recheck your most basic assumptions : are you using the computer you think you are using? are you testing what you think you are testing? are you observing what you think you are observing?
  • Eyewitness reports leave out a lot of potentially vital information. So listen, but DO NOT BECOME ATTACHED to the claims people make .
  • Invite more observers and minds into the investigation.
  • Create incentives for people to report intermittent problems.
  • If someone tells you what the problem can’t possibly be, consider putting extra attention into those possibilities.
  • Check tech support websites for each third party component you use. Maybe the problem is listed.
  • Seek tools that could help you observe and control the system.
  • Improve communication among observers (especially with observers who are users in the field).
  • Establish a central clearinghouse for mystery bugs, so that patterns among them might be easier to spot.
  • Look through the bug list for any other bug that seems like the intermittent problem.
  • Make more precise observations (consider using measuring instruments).
  • Improve testability: Add more logging and scriptable interfaces.
  • Control inputs more precisely (including sequences, timing, types, sizes, sources, iterations, combinations).
  • Control state more precisely (find ways to return to known states).
  • Systematically cover the input and state spaces.
  • Save all log files. Someday you’ll want to compare patterns in old logs to patterns in new ones.
  • If the problem happens more often in some situations than in others, consider doing a statistical analysis of the variance between input patterns in those situations.
  • Consider controlling things that you think probably don’t matter.
  • Simplify . Try changing only one variable at a time; try subdividing the system. (helps you understand and isolate problem when it occurs)
  • Complexify . Try changing more variables at once; let the state get “dirty”. (helps you make a lottery-type problem happen)
  • Inject randomness into states and inputs (possibly by loosening controls) in order to reach states that may not fit your typical usage profile.
  • Create background stress (high loads; large data).
  • Set a trap for the problem, so that the next time it happens, you’ll learn much more about it.
  • Consider reviewing the code .
  • Look for interference among components created by different organizations .
  • Celebrate and preserve stories about intermittent problems and how they were resolved.
  • Systematically consider the conceivable causes of the problem (see below).
  • Beware of burning huge time on a small problem. Keep asking , is this problem worth it?
  • When all else fails, let the problem sit a while, do something else, and see if it spontaneously recurs.

That's a lot of critical thinking required. Do you know a lot of institutions that love and value critical thinking and critical thinkers? No? And that's a lot of institutional savvy required. Do you think that CMS and HHS -- even with Obama backing them[3] with all his lameduck political clout -- are going to be able to get institutional cooperation from all these players on a crash basis? I doubt it, especially after the Obama public relations machine told everybody "Relax! It's just a soft launch!" and "The only date that really matters is January 1!"

From 07/05/2013, here are the institutions whose systems must be integrated on the backend of the Federal Exchange:

So what's the difference? The system architecture. Medicare has a simple and robust single payer architecture: You determine eligibility in one (1) jurisdiction (the United States) with one eligibility criteria for citizens: Their age. ObamaCare, by contrast, needs to determine eligibility in 50 (fifty) jurisdictions [for ObamaCare as a whole; 36 for the Federal Exchanges], with a complex eligibility formula that's primarily income-based, but involves systems integration from the IRS, DHS, HHS, and private credit reporting companies (at least), to throw people into the right subsidy bucket. That's called a combinatorial explosion, and even the best program and project managers -- which ObamaCare's managers clearly either are not, or have not been given the opportunity to be -- have a hard time dealing with them. Let me know how it all works out....

Every single one of those institutions owns at least one system that needs to be integrated into the Exchanges, and each is a potential source of intermittent #FAIL. (That's before we get to the combinatorial explosion of #FAIL caused by interactions between systems.) And remember, if one component fails, the system fails. Intermittently. Suppose every applicant on the Exchange has to be checked to see if they're a Native American; fact, they do. It doesn't matter if DHS give back its results slick as a whistle if the Bureau of Indian Affairs has intermittent failures. So technical issues aren't the main story here; the truly challenging issues are never technical. Rather, we have management issues:

  1. Political: Wrangling IRS, DHS, HHS, private credit reporting companies, and IIRC the Bureau of Indian Affairs, and the Peace Corps[4];
  2. Personnel: Finding critical thinkers and deploying them, in whatever institutional settings they are to be found.

Now, in a perfect world, there are just a few bugs, and by Monday, the Federal Exchange will be up and running, and sticking out your arm for health insurance parasites to sink their tiny, rent-extracting mandibles into buying health insurance online will be as easy as buying a flat-screen teebee or a plane ticket. I'm guessing no, because that greybeard's phrase, "intermittent failure" should give you the heebie jeebies, as it did me. You don't fix that stuff in a weekend.

So, Kremlinology: If Obama appoints a Czar to fix the Federal Exchanges, we're in "buff the turd" mode. That will mean the White House has determined that political and personnel problems with the Federal Exchanges are insoluble.[5]

NOTE A word on "the government can't write software." The government can write brilliant software: How do people think the Mars Landers are run? Tin cans and string? The question that should be asked is: Why was the government tasked with writing this software, when the rugged, robust, simple and proven single payer architecture was available to them? Jon Benteley: "The cheapest, fastest, and most reliable components are those that aren't there." Throw the requirement to preserve rental extraction by health insurance agencies out of the equation, and everything becomes clean. And implementable. Could have been done in a year, as LBJ rolled out Medicare in a year. Whoever wrote the requirements for this thing should be publicly shamed.

NOTE I don't have the time to add this part, but check out this link for the cultural differences between frontend and backend developers. Note that the famous Reddit thread is frontend developers. More management challenges! And here's a piece of what turns out retrospectively to have been frontend-centric hagiography back in June.

NOTE [1] Back in the day when I was a fancy pants consultant, I would hear people say "I don't want my software built by people who don't have kids." Unfair perhaps -- FaceBook, Google -- but those projects often had lives at stake. As does ObamaCare. I guess, to avoid offensive -- justly offensive -- age-ism, I'd say better to have old heads on young shoulders. Developers like that do exist, fortunately. Oh, there's a picture of the author at the link. Yes, he has a grey beard. He knows the Great Runes!

NOTE [2] Frontend vs. backend from Stack Overflow:

Generally speaking, the "front end" is the user interface, and the "back end" is the code supporting that front end (responsible for database access, business logic etc). ...

Frontend is what you do that the user can see. Like designing a user interface. Backend programming is the code that the user doesn't 'see'. This is what works with the data behind the scenes. For example fetching/inserting/deleting/updating a database. ...

So, on the blog you're reading, everything that you as reader or contributor see in the browser, without exception, is the frontend. (Admins control the backend: In essence, in the world of Drupal or WordPress or whatever, the database, the server, and the operating system on which they run, including the file system.) When you press Submit, your post on the front end is sent over the Intertubes and stored in by Drupal or WordPress or whatever in the blog's database, which is why you can retrieve it the following day, which could not happen if your post were stored in the browser, right? Similarly, every time your post is displayed, Drupal or WordPress or whatever is grabbing your post from the database, formatting it into HTML, and sending the HTML over the Intertubes to somebody's browser, maybe yours, along with the sidebar, the footer, the logo, and so forth, all of which is also, ultimately, stored in the database. (Well, JPGs and so forth are stored on the file system. And I'm leaving out caching. And clever stuff JavaScript, which can also connect to the server, can do.)

The Federal Exchanges work using the exactly the same system architecture, although on a much bigger scale. Simplifying horribly: The process of setting up an account, for example: You enter data on the front end in a form. You press submit. Your data #1 goes over the Intertubes, and the server grabs it and stores it in a database on the backend, and #2 the backend sends back a confirmation to the browser, again through the server over the Intertubes, that your data has been stored and that it's OK for the browser to go ahead to the next screen. Imagine a world where the front-end developers had created a frontend of a Jony Ives-like slickness. The backend has at least two points of failure at #1 and #2. Suppose at #1 the database is down. That would look like a failure of the front end to respond ("It's hung up!") but it's really a backend thing. Suppose the server at #2 died, as servers will do. Again, the frontend would look like it died, but it's just helplessly waiting for the confirmation code, which hasn't arrived. The intermittent failures could happen either at #1 or #2.

Somebody really technically competent who does this for a living would laugh at the paragraph above, since the blog you are reading is basically a Wright Brothers Flyer compared to the Federal Exchanges, which are more like a Boeing 747 (or perhaps a Spruce Goose). Nevertheless, both sites and both airplanes share a fundamental architecture and fundamental engineering principles, which you must understand to cut through the bullshit and the bafflegab of what's going on with the Exchanges, technically.

NOTE [3] Remember, it's highly likely that the White House totally borked the entire project by changing the forms -- that is, the stuff you fill in on the Exchange website -- at the last minute for political and public relations reasons. Speculating freely: If the ObamaCare Exchange's front end, where the form is filled out, was tightly coupled to the back end -- that is, if the backend expected to find some chunk of data in exactly one place in the form, from where it would grab it and slam it directly into the database on the backend, that's quite likely. That's the quick and dirty way a body shop under pressure would do the job, and we all know the Obama administration would never cut corners on something that didn't directly impact public relations, right? Ha ha. Anyhow, we do know that changing the forms broke the Connecticut, state-based Exchange, so it's reasonable to think that the Feds had the same problem. Except for 36 states, not one. And that means that Obama is going to people he screwed over to ask them for a shovel to dig himself out of a jam. To technical people, this is standard operating procedure, but the managers and department heads may not be so compassionate.

NOTE [4] "It"s all gone political, sir." Terry Pratchett, Thud.

NOTE [5] I missed the obvious one: Privatize it all. The market can never fail. It can only be failed. If Obama actually managed this process so that privatization of the Federal Exchanges is the outcome, my hat is off to him; he's either lucky as only the truly evil can be, or he really can play 11-dimensional chess.

No votes yet


Rainbow Girl's picture
Submitted by Rainbow Girl on

From a comment at Naked Capitalism:

sleepy says:
October 5, 2013 at 9:57 am
RE: Obamacare startup

I’ve tried to get an account going to check on what’s available and at what price since Tuesday.

I finally succeeded yesterday after switching browsers from firefox to explorer and creating a new email account at yahoo.

I’m good to go, right?

Nope–I plug in all the info, and a popup appears:

There was an error in establishing a connection with Experian. Please contact Experian at ————-”.

Yes, I knew the credit folks were involved, but I’m supposed to contact them myself to tell them, what, that the federal exchange can’t reach them?


How about that? Whatever integration exists for data sharing-moving *between* the army of agencies (public and private) involved in data mining/collecting for ObamaCare, it doesn't seem to apply to the marks (customers). Now the prospective enrollee can look forward to making phone calls to (depending on the piece of the puzzle that's a #FAIL): credit reporting agencies (up to thre!); DHS, IRS, HHS, CMS, ICE, Bureau of Indian Affairs, [fill in -- who did I leave out?][/fill]

I'll bet those Young Invincibles with their habits of instant-everything-online will *love love love* this!

NOTE. Does this system of having the system-user make calls to whatever entity is causing the #FAIL emblematic of body-shop work? In any event, it takes cost-shifting to a whole new level in terms of stealing people's TIME as well as their money.

Submitted by lambert on

1. Experian is a credit reporting agency. So we have a public-private partnership in #FAIL

2. Excellent example of issues in our provenance discussion:

a) To verify the quote, I needed a link to the comment at NC. So I searched for it, found it, and used my awesome administrative super-powers to add it too your comment

b) This is a classic case of where a screen dump would be invaluable; see discussion here.

3.) The fact itself.

Rainbow Girl's picture
Submitted by Rainbow Girl on

"To verify the quote, I needed a link to the comment at NC."

Oops, mea culpa. I'm sorry I created extra work for you but I'm glad you have those awesome superpowers!

(Maybe a reply to the poster on this incident re doing a screen dump ... if he/she can replicate his/her experience)

Dromaius's picture
Submitted by Dromaius on

Yes, I knew the credit folks were involved, but I’m supposed to contact them myself to tell them, what, that the federal exchange can’t reach them?

Oh, please, people on federal Exchanges, call Experian when you get this error! (I said with prayer hands and pleading eyes). Bombard their phones. Please.

(I wanted to cry, so I laughed instead).

Rainbow Girl's picture
Submitted by Rainbow Girl on

Bombard their phones

Unfortunately Experian and the other two members of the credit reporting Monopoly (yes, common theme, see, e.g. Insurance Industry in US is a non-organic cartel/monopoly) have that covered. Every try calling Experian or Equifax or that other one? Phone trees! Disconnection after 20 minute hold! Nonworking number! They are impregnable to direct consumer contact.

On the other hand -- people *could* start lodging complaints with the CFPB, which has Experian et al squarely in its jurisdiction. Sample complaint: "Experian is preventing me from enrolling in ObamaCare." That should get CFPB's (and WH's) attention!