It's time to stop advocating the use of the wrong model in clinical chemistry - 3/2004

Introduction

A recent article in Clin Chem News described instrument evaluation methods (1). Surprisingly absent from this article was any reference to evaluation methods that measure (vs. model) total error, especially since many of the methods cited in this article are NCCLS standards and there is an NCCLS standard specifically devoted to measuring total error (2). This editorial will discuss some of the implications of this deficiency and the use of incorrect models.

The Importance of Diagnostic Assays

When I started in the field over 20 years ago, I asked several clinicians how they used diagnostic assay results. A common answer was “to confirm a clinical diagnosis.” These days that answer has changed for many clinicians to, “to establish a clinical diagnosis.” Putting things another way, a test result can be the prime reason that a clinician recommends a further diagnostic procedure such as a biopsy, a certain treatment, or to withhold treatment. This puts a great burden on manufacturers, laboratorians, and regulators to ensure that diagnostic assay results are as free from error as possible. As an example, take a case that appeared in the media (3) about a woman who had repeated elevated hCG results and was treated for cancer. The hCG results were falsely elevated due to HAMA interference and the woman had no cancer. Moreover, this was not an isolated case – it had occurred repeatedly (4).

Measuring quality vs. modeling quality

There are two possible approaches to determining the quality of diagnostic results: measuring or modeling.

Measuring has the advantages of being simple, requires no assumptions (other than a representative sample), and requires no model, and models can be wrong.

Measuring has the disadvantages of requiring more samples to give the same confidence as modeling (assuming of course that the modeling is correct). Measuring also does not provide causes for observed errors.

Modeling has the advantage of providing a cause for errors. This is crucial for quality improvement.

Modeling has the disadvantage of requiring the model and its assumptions to be correct. Whereas this may be an obvious requirement, it is difficult to prove that a model and its assumptions are correct. Whereas modeling provides more information than measuring, it also requires considerably more effort.

Measuring methods

Measuring methods are described by the NCCLS standard EP21 (2) or a recent review (5). Basically, one assays a series of patient specimens by a candidate and reference assay. Calculation of the total error is by a parametric or nonparametric method.

Modeling methods

Now here is where there is a major problem (6). There are several well known clinical chemists who propose that average bias plus imprecision equals total error. Some of these authors suggest that:

• if you have a specific requirement for medical error
• follow their quality control scheme
• you will achieve the desired quality

It just isn’t true. These authors ignore a more complete model of an assay (7) which takes into account the random interferences that often occur in patient samples. Moreover it is just these random interference errors that

·        cause clinical problems because the errors are so large and

·        they cannot be detected by quality control

The reason that these problems can’t be detected by traditional quality control is that the model associated with quality control does not (and cannot) deal with these interferences. Continually advocating a wrong model perpetuates its use by others - as an example see (8) - and has the potential to retard quality improvement.

The companies for which I have worked used more complete models such as that described in reference 7 but manufacturers typically don’t provide to the public their evaluations methods. Not all laboratorians use the wrong model. A recent article about cholesterol compared the measurement method with the wrong model (9).

Outliers

Finally, a word about outliers. The word outlier is somewhat unfortunate because outliers often seem to get special treatment as in “oh don’t worry about that result, it’s an outlier.” Outliers might better be thought of simply as errors, albeit large errors. In a measuring method, outliers are not discarded whereas in modeling methods they are often discarded. Yet in real life they appear.

Conclusions

One can always simply measure the errors in an assay (e.g., without modeling). If one chooses a modeling method, then it’s time to use a more complete model and abandon the model that’s been shown to be wrong.

References 

1. Nichols JH Instrument Validation The Road to Success. Clin Chem News 2004;30:2,14-16.  
2. National Committee for Clinical Laboratory Standards. Estimation of total analytical error for clinical laboratory methods; approved guideline. NCCLS document E21-A 2003 NCCLS Villanova, PA.
3. Sainato D. How labs can minimize the risks of false positive results. Clin Chem News 2001;27:1,6-8.
4. Cole LA, Rinne KM, Shahabi S, Omrani A. False positive hCG assay results leading to unnecessary surgery and chemotherapy and needless occurrences of diabetes and coma. Clin Chem 1999;45:313-314.
5. Krouwer JS. Setting performance goals and evaluating total analytical error for diagnostic assays. Clin Chem 2002;48:919-927.
6. Westgard JO, Petersen PH, and Wiebe DA Laboratory process specifications for assuring quality in the U.S. National Cholesterol Education Program. Clin Chem 1991;37:656-661. There are many other references as well.
7. Lawton WH, Sylvester EA, Young-Ferraro BJ. Statistical comparison of multiple analytic procedures: application to clinical chemistry. Technometrics 1979;21:397-409.
8. Boyd JC and Bruns DE Quality Specifications for Glucose Meters: Assessment by Simulation Modeling of Errors in Insulin Dose Clin Chem 2001;47:209-214.
9. Miller WG, Waymack PP, Anderson FP, Ethridge SF, Jayne EC. Performance of four homogeneous direct methods for LDL-cholesterol. Clin Chem 2002;48:489-498.