Krouwer Consulting
 

Home
Up
Modeling
Freq. of Med. Errors
Frequency of Medical Errors II
Six Sigma
Six Sigma II
Six Sigma III
Outliers
Equivalent QC
Equivalent QC2
Sys. not People
Near Miss
Detection
Recovery
Risk Management
FMEA FRACAS
Fault Tree Example
Beware the Technical Administrator
Discrepant Analysis
FMEA Validation
Proficiency Testing and Six Sigma
It's Up To the Lab Director
Bland Altman Plots
Risk Management II
Unit Use QC
The reverse Pareto – not a good
Risk Management III
Pedigree of Approaches to Reduce Error
What it Takes to Get New Ideas Accepted
FMEA FRACAS FTs Pareto
When to conduct FMEA
Error Reporting Systems for Clin
Customer Misuse
Fault Isolation
Uniformity of Claims
Outliers in QC and Proficiency Testing
The quality of quality initiativ
Publishing in journals or on web
FDA Waiver Guidance
Pay for Performance
Pay for performance II
Performance vs. Attribute Goals
ISO Terminology
Latent Errors
Preventable medical errors
Troponin I
GUM Comment
More on GUM
Standards Issues
Slow Qual. Progress
More QC?
Report Writing
Software Validation
Dump Data

Pay for performance II

Two regulatory questions that are asked for procedures are:

1.  to approve or reject, and

2. how to improve approved procedures

To approve or reject

Consider a hypothetical case in which there is a disease with a mortality rate of 20% for persons who go untreated. Treatment, while efficacious, has a yearly average mortality (1) rate of 0.05%. A rational decision for a person who has the disease is to elect for treatment since the expected value of surviving treatment is much higher than surviving no treatment. But based on a million treatments, 500 people will nevertheless die each year due to treatment.

This raises some questions on how regulatory agencies should set quality goals and decide what is approved and what is not. However, an immediate problem can be seen for this hypothetical case, since one can argue that the treatment failure rate could be close to 20% because for any failure rate up to 20%, treatment will result in less mortality than no treatment, even if all treatment failures are due to preventable medical errors. Of course, there would be a public outcry were preventable medical errors that high and pressure would be exerted on regulatory agencies for improvements.

To summarize the tradeoffs:

One has to quantify these tradeoffs (for example, using decision analysis principles) to arrive at the most favorable mortality outcomes. This is also complicated by the fact that high profile errors will weigh more on regulators than low profile errors (e.g., wrong site surgery gets more attention than infections).

Note that this is not a complete picture in that there are many cases, for which approval has not been sought. This could be caused by many reasons including the expected return on investment for the treatment vs. the expense of filing for approval. This leads to off label uses of drugs (2) and diagnostic assays. In these cases, the clinical community, with or without scientific evidence has rendered an approval decision, albeit without any regulatory input.

How to improve an approved procedure

Consider some data to go with hypothetical example above and assume that although the treatment is approved, a regulatory body can give or withhold permission for a hospital to perform treatment.

Hospital mortality rates for a procedure

Assume that additionally, based on the analysis of these errors, one could expect improved processes to have no more than 5 deaths per year (0.0005% mortality rate). This is shown by the arrow in the figure.

Note that when regulatory agencies grant or not grant permission to allow the treatment, this is a dichotomous decision. For example, although the Joint Commission often gives out a variety of accreditation “grades” one can always dichotomize all grades into two categories – pass (allow the hospital to operate) or fail (shut down operations). So what are some possible goals?

Set the goal at 1,000 – The benefit of this goal is that the distribution of outlier hospitals will be prevented from performing the procedure. Assuming that no treatments will be withheld (e.g., all people desiring treatment will be accommodated elsewhere, this goal improves mortality. But one problem with this goal, is that for the hospitals that pass, there is no incentive to improve.

Set the goal at 5 – The problem with this goal has been discussed. The treatment will be unavailable and the mortality will increase by a huge amount. As a diagnostic assay example, consider a new goal for analytical performance of home glucose testing that can’t be met by manufacturers leading to taking all products off the market. An increase in mortality would follow.

Set the goal somewhere between 250 and 1,000 – This goal also has potential problems. At some point for a goal that approaches 250, too many hospitals will be prevented from performing the treatment, such that some persons desiring treatment won’t be able to get it and mortality will increase. At a point closer to 1,000, it is possible that these hospitals would improve their processes, which would shift the curve to the left, but this will be a difficult goal to set.

Set the goal on some other conceptual basis – This has been proposed for diagnostics assays using a variety of concepts (3). I would argue that however rational these concepts are, they do not compare with the rationale of using the actual performance data to set goals, which in turn is used because the real goal is – does the treatment (or information provided by a diagnostic assay) lead to improved mortality compared with the alternative of withholding treatment (or rejecting approval of the diagnostic assay).

The pay for performance rationale

The benefit of a pay for performance program is that one can set a goal at 1,000 to immediately prevent exceptionally poor performers, but use pay for performance to provide an incentive for accredited hospitals to improve. As improvement occurs, and the curve shifts to the left, one can reset the 1,000 goal as well as pay for performance incentives.

I have critiqued some aspects of pay for performance as follows:

One must consider all errors for all procedures, but pay for performance programs have developed only a small set of measures. This set of measures may not be optimal and even if there were an optimal set, it might change more quickly than measures are set. An alternative is to use a FRACAS (Failure Review And Corrective Action System) whereby all errors are counted and ranked for priority.

Also some proposed solutions violate the dynamic nature of FRACAS, in which one is going through many cycles of implementing changes to a procedure, measuring progress, and making new changes. For example, to reduce wrong site surgery, the Universal Protocol is required, but it has been shown that this by itself would not prevent all wrong site surgeries (4).

Other concerns with pay for performance

It has been suggested that pay for performance could reward hospitals with a higher baseline performance rather than for improvement (5).

Centor expresses similar concerns that I have previously had, but does so with a (hypothetical) medical example (6).

References and notes

1. To simplify, the word mortality is used. Alternatively, one could replace this with morbidity and mortality to signify a range of adverse effects.
2. Radley DC, Finkelstein SN, Stafford RS Off-label Prescribing Among Office-Based Physicians Archives of Internal Medicine 2006; 166: 1021-1026.
3. Fraser, CG. Biological Variation: From Principles to Practice 2001, AACC Press, Washington DC.
4. Kwaan MR, Studdert DM, Zinner MJ, Gawande AA Incidence, patterns, and prevention of wrong-site surgery. Arch Surg. 2006;141:353-7
5. Rosenthal MB, Frank RG, Zhonghe L Epstein  AM Early Experience With Pay-for-Performance JAMA. 2005;294:1788-1793.
6. Robert M. Centor, MD; Michael S. Barr, MD Point/Counterpoint on Pay-for-Performance, see http://www.medscape.com/viewarticle/528570