Risk Management – Improving lab quality, or the latest fad - 8/2006

Long a part of other disciplines, risk management has become very visible in healthcare in the last few years. Perhaps the biggest reason is that since 2002, the JCAHO (Joint Commission on Accreditation of Healthcare Organizations) requires healthcare organizations to conduct at least 1 FMEA (Failure Mode Effects Analysis) each year. Since then, more attention is being paid to risk management in healthcare. For example, the organization CLSI (Clinical and Laboratory Standards Institute) devoted its 2006 annual meeting to this topic. For diagnostic assay manufacturers, risk management was on the map earlier since starting in 1996, FDA has required manufacturers to perform risk management activities through QSRs (Quality System Regulation).

This essay covers various dimensions of risk analysis as shown by the following figure.

Risk management standards – horizontal vs. vertical

Standards are a good place to start since standards should contain guidance about risk management.

There is an ISO standard (14971) about risk management for medical device manufacturers. (There is an appendix – it’s called an annex in ISO speak – for manufacturers of diagnostic assays). This is a so called “horizontal standard” meaning that principles are explained but unlike a “vertical standard” no detailed specific procedures to follow are recommended such as those – for example - in the CLSI standard EP9 (how to perform a method comparison to estimate bias). Horizontal standards are often called “flexible” with the appeal that one can pretty much do anything and say that one is conforming to the standard. So vertical standards would be useful for risk management.

There is a CLSI standard EP18A concerned with FMEA for unit use devices. This is also pretty much a horizontal standard.

Risk management goals – qualitative vs. quantitative

Goals have been hard to come by in lab medicine. Often, lab goals are more qualitative than quantitative as in “we don’t want to release any problem results.” One possible set of quantitative goals might be answers to the questions:

1.       what difference from reference constitutes an unacceptable error and

2.       how many times is that difference allowed to occur.

Although much have been written about the topic, there are few standards. In fact, two groups gave up on this task (CLSI EP20 and ISO 15196). There is an ISO standard (15197) for home use glucose assays, which answers both questions. That is, medical acceptability limits are given but the rate required for acceptable results is only 95%. This means that up to 50,000 medically unacceptable results per million results would meet the ISO goal. Clearly, something is wrong which was described in an article about goals (1) which suggested that along with limits given by the glucose standard, (allowable total error), there needs to be another wider set of limits for which no or few results are permitted. Note that the new proposed waiver guidance from the FDA (see related essay) embraces this concept.

Levels of risk management

Anyone can write software, yet in the software industry, the level of expertise of an organization to write software has been formalized through the Capability Maturity Model Integration (2) from Carnegie Mellon. So here is an attempt to categorize risk management practices in a similar fashion.

1. no formal activity
2. does the minimum to meet regulatory requirements, largely a documentation exercise
3. department initiated activities that follow established risk management procedures and contain progress measures
4. quantitative use of fault trees, fault trees updated as needed and risk management is an integral part of the process that delivers service

From what I see, most of the diagnostic assay industry is at level 2 and sometimes approaching level 3. Labs are transitioning from level 1 to level 2. Nuclear power is an example of an industry at level 4. A recent article (3) about risk management for medical devices describes fault trees in terns of level 4, although it is doubtful that this level of fault trees are carried out.

The issue is that saying one is conducting risk management can mean anything from level 1 to level 4.

Knowledge of risk management

In my experience in clinical chemistry, I often see people with a limited knowledge of risk management involved in risk management activities. To assess your interest/knowledge, take the risk management quiz

 

1. What is the annual RAMS meeting?
2. What is a minimal cut set? (hint: think fault trees)
3. Have you ever attended your local IEEE Reliability meeting?

Answers

1. RAMS (Reliability and Maintainability Symposium) is comparable to the AACC annual meeting for clinical chemists as the most important meeting in reliability (which includes risk management).
2. A minimal cut set is used in quantifying fault trees. A cut set is said to be a minimal cut set if, when any basic event is removed from the set, the remaining events collectively are no longer a cut set. (A cut set is a collection of basic events; if all basic events occur, the top event will occur.)
3. Click here a list of local IEEE sections

Potential vs. observed errors

Any process has both:

• potential errors (errors have not yet occurred)
• observed errors (errors have occurred)

Risk management can be thought of as addressing both types of errors. FMEA addresses potential errors and FRACAS (Failure Reporting And Corrective Action System) addresses observed errors. FRACAS is not mentioned as a technique in ISO 14971 but consider the case where errors have been observed. If one puts in place effective corrective actions, one has reduced the risk that these errors will recur. This is an important case for manufacturers, who often exercise their instrument system during development to expose and correct for errors (4).

Risk management within the culture of an organization

I give a training session on risk management which has a section which starts with “I have never met an engineer who wanted to be in FMEA meeting”. There are many reasons why this is so including:

• engineers “signed up” to design things – not do FMEAs
• lack of management commitment
• lack of resources

Risk management activities are at times conducted during lunch, with lunch provided as an attendance enticement, but also a signal that the risk management activity shouldn’t interfere with normal work activities. Moreover, risk management activities are often conducted to meet regulatory requirements and are conducted once. The goal of such programs is to pass an inspection or audit.

When a department initiates its own risk management program, it usually has quantitative, measurable goals and is often carried out over a longer time period for the product or process. Success is defined as meeting these measurable goals. To summarize:

Risk management programs

Risk management possibilities

Risk management has the potential to improve quality and reduce error rates. Yet risk management practices can range from almost no activity to extensive programs. Benefits from risk management will be proportional to the effort that is expended.

References

1. Krouwer JS. Setting Performance Goals and Evaluating Total Analytical Error for Diagnostic Assays. Clin. Chem., 48: 919-927 (2002)
2. See http://www.sei.cmu.edu/cmmi/
3. Snow A. Integrating Risk Management into the Design and Development Process, Medical Device & Diagnostic Industry 23, no. 3 (2001): 99–111, available at http://www.devicelink.com/mddi/archive/01/03/002.html
4. Krouwer JS Using a Learning Curve Approach to Reduce Laboratory Error, Accred. Qual. Assur., 7: 461-467 (2002)