FMEA / FRACAS / RCA vs. Analytical performance – the big difference in lab quality goals - 7/2005

Historically, laboratorians are used to performing quantitative method evaluation studies such as evaluation of bias and imprecision and monitoring performance with quality control and are only now starting to think about how to prevent other errors by using methods such as FMEA, FRACAS, and RCA.

FMEA = Failure Mode Effects Analysis
FRACAS = Failure Review And Corrective Action System
RCA = Root Cause Analysis

First, an explanation of some different error types that one might encounter in a lab

1. Attribute (discrete) errors – example: a person mislabels a sample which leads to an assay result being reported which does not belong to that person. These are discrete variables; that is, the variable patient id mix-up either occurs or doesn’t occur each time a sample is processed and has a rate of occurrence = numbers of errors divided by number of tries.
2. Analytical performance errors – example, a sodium assay has a 6% average bias to reference. These errors are continuous variables – the bias found can be any number, unlike the discrete case which is usually represented by 1 or 0.

Although not really a separate category, note that attribute errors can occur as causes within analytical performance evaluations. For example, if there is a hardware problem on an analyzer that causes a sodium to be reported (without any error flag) as 160 mmol/L, instead of 140 mmol/L, this error might be discarded from a performance evaluation as an outlier. The attribute error is the cause of the performance error.

Note that the CLSI (formerly NCCLS) document EP21A allows performance errors to be treated as attribute data (1). That is one can simply count the number of occurrences that assay data fails (for any reason) performance goals. The value of this is that one can use FMEA, FRACAS, or RCA and include performance issues.

Consider goals for each type of error.

Analytical performance goals – The ISO glucose standard (15197) contains a performance goal. According to this standard, the “minimum acceptable accuracy” goal is:

“Ninety-five percent (95%) of the individual glucose results shall fall within ± 0,83 mmol/L (15 mg/dL) of the results of the manufacturer’s measurement procedure at glucose concentrations < 4,2 mmol/L (75 mg/dL) and within ± 20 % at glucose concentrations >= 4,2 mmol/L (75 mg/dL).”

As described in the essay on Six Sigma, for an assay that just meets requirements this means that just under 5% of the values could exceed the goal which means that the number of defects one will see is 50,000 defects per million assays! In six sigma terms, this is close to a 3 sigma process (3.1). Remember that as defined by the ISO standard, defects are based on failing to meet medical requirements.

There are several reasons why this situation doesn’t lead to more of an outcry.

1. In spite of the goal, the actual rate of defects may be lower
2. There is a continuum of quality, e.g., a error of 19.9% and 20.1 are very similar even though the 19.9% error is within goals and the 20.1% fails. So there may be a bunch of values just outside of the goal.
3. Lab errors often are not the sole cause of adverse patient events, e.g., the clinician may not believe the result and request it to be rerun. In fault tree terminology, lab errors are often "AND" events. Other events must also occur for the effect of the lab error to be realized.
4. Lab errors are often hard to trace as a cause for an adverse patient event

Reason #2 is discussed in more detail in the outlier essay. That is, if one added to the glucose goal, outlier rates which could not be exceeded, then one would have a more complete and improved goal. Ideally, the outlier rates would be conditioned by use of a Parkes grid (2). Effectively, one would have largely converted the continuous variable "glucose errors" into a series of discrete (attribute) buckets.

Reason #3 is particularly important to consider since this means that even for the worst glucose performance errors (e.g., severely hyperglycemic reported as hypoglycemic or vice versa), there is no data that I am aware of that provides the frequency of death or injury for this lab error. However, even if death or injury do not occur, one could classify this error as a dangerous near miss (e.g., the effect of the error is prevented by chance (e.g., unplanned) detection.

One could ask, why have I chosen a home use glucose goal instead of a lab goal, since this topic is about lab goals. The problem is that unless, the goal is expressed as a rate, the goal is not really useful and also do not conform to other goals used for medical errors in healthcare. So this is the closest goal that I could find. There are quite a few papers on diagnostic assay goals but many of these don't express goals in terms of rate and others are not consensus based standards. I have previously commented on the inadequacy of a cholesterol goal (3).

Attribute performance goals - Contrast the above performance goal with the VA (Veteran’s Administration) criteria for medical error frequency (4).

Frequent - Likely to occur immediately or within a short period (may happen several times in one year)
Occasional - Probably will occur (may happen several times in 1 to 2 years)
Uncommon - Possible to occur (may happen sometime in 2 to 5 years)
Remote - Unlikely to occur (may happen sometime in 5 to 30 years)

The VA severity criteria are as follows for patient outcomes (see reference 2 for other categories).

Catastrophic Event - Patient Outcome: Death or major permanent loss of function (sensory, motor, physiologic, or intellectual), suicide, rape, hemolytic transfusion reaction, Surgery/procedure on the wrong patient or wrong body part, infant abduction or infant discharge to the wrong family
Major Event - Patient Outcome: Permanent lessening of bodily functioning (sensory, motor, physiologic, or intellectual), disfigurement, surgical intervention required, increased length of stay for 3 or more patients, increased level of care for 3 or more patients
Moderate Event - Patient Outcome:  Increased length of stay or increased level of care for 1 or 2 patients
Minor Event - Patients Outcome: No injury, nor increased length of stay nor increased level of care

So with this VA classification with 4 being worst and 1 being best, the VA has a severity by frequency grid where each cell has frequency multiplied by severity. The shaded cells are high priority cases.

The glucose performance standard goal doesn’t even fit on the VA table! If one interprets the highest VA frequency (may happen several times in one year) as 3.4 times per year, then the VA starts with 6 sigma as undesirable in all but cases of minor severity and recommends a lower frequency of occurrence – as low as once in 5-30 years! On the other hand, if the glucose performance goal is representative of other performance goals and a lab reports one million results per year, then the performance goal is < 50,000 medically unacceptable performance errors per year.

The VA (and other thought leaders) thinking about attribute type error frequency should help laboratorians realize that their world of performance goals is out of step with the general trends in reducing medical errors.

References 

1. Estimation of Total Analytical Error for Clinical Laboratory Methods; Approved Guideline NCCLS EP21A, NCCLS, 771 E. Lancaster Ave. Villanova, PA., 2003
2. Parkes JL, Slatin SL, Pardo S, and Ginsberg BH. A new consensus error grid to evaluate the clinical significance of inaccuracies in the measurement of blood glucose. Diabetes Care 2000;23:1143-1148.
3. Krouwer JS. Problems with the NCEP (National Cholesterol Education Program) Recommendations for Cholesterol Analytical Performance. Arch Pathol Lab Med 2003;127:1249.
4. The Basics of Healthcare Failure Mode and Effect Analysis, available at http://www.patientsafety.gov/SafetyTopics.html#HFMEA