Krouwer Consulting
 

Home
News
Products
Services
Blog
Essays
Publications
Search
Links
About

Medical error quality blog

This blog contains selected rants and other items on all aspects of the effort to reduce medical errors. If you send me a comment, I will post it in the appropriate section. Also, you can optionally add a reference to your web page or email address.

Important Note - See also my Word Press blog, where there are more comments.

8/04/08 - CLSI EP22 EP23 Review - Update

EP22 was created as a means to use risk management to allow manufacturers to recommend the frequency of external quality control run by clinical laboratories. This was the so called option 4. Options 1-3 were part of the original CMS proposal to allow clinical laboratories to reduce the frequency of external quality control to once a month (provided certain conditions were met).

EP23 was the clinical laboratory follow on document to EP22.

Here’s my take on these two documents.

1.       Manufacturers won’t provide the information as suggested by EP22.  (This information consists of experiments to demonstrate the efficacy of internal control measures). It would be a lot of work (e.g., cost) and there’s no regulatory requirement to do so. Moreover, if this information were provided, then it is labeling which would require FDA to review it. It is not clear that FDA has accepted this review task.

Update on 8/4/08 - During a CLSI presentation at the AACC meeting in Washington, Alberto Gutierrez from the FDA gave a presentation. Afterwards, I asked him if FDA would review the material about internal control experiments that manufacturers might present as part of the package insert. He said that FDA would review this material - but from what was said it seemed that the review would be superficial and that only egregious problems would be flagged by the FDA.

2.       Clinical laboratory staff does not have the expertise to review this information, were it provided. This does not mean that clinical laboratory staff is incapable of reviewing it – they could acquire the expertise – it just seems unlikely.

3.       Should manufacturers provide this information and clinical laboratory staff review it, there would be no benefit with respect to improving QC. This is illustrated by an example in EP22 where the failure mode of “incorrect results due to low volume sample” is examined. After presenting the results of an experiment to show how an internal system control works, the user control measure is to “ensure that adequate volume of sample is presented to instrument.” But clinical laboratory staff would (or should) do this anyway. They don’t need EP22 and EP23 to know that one should follow the manufacturer’s instructions and to refrain from doing something stupid.

In clinical chemistry, risk management is “in.” But there are signs that its popularity is already starting to wane. This is unfortunate, as there is a great opportunity to use risk management tools to reduce both the risk and occurrence of laboratory errors. But one must focus not just on potential system errors, as EP22 and EP23 do, but on human errors as well.

6/17/08 - Reading Quality Digest can be dangerous to your health

In the June 2008 issue of Quality digest, there is an article by Jay Arthur entitled “Statistical Process Control for Healthcare”. After the usual boilerplate type of introduction, something caught my eye; namely, the so called good news that there is “inexpensive Excel based software to create control charts … .“ This made me go to the end of the article where sure enough the author just happens to sell such software. This may have been a good place for the author to introduce the term bias.

To understand a more serious problem with this article, consider a hospital process; namely analyzing blood glucose in a hospital laboratory. Because such a process has error, quality control samples are run. Say such a control has a target value of 100 mg/dL.  The values of the quality control samples are plotted by SPC software and rules are formulated. If the glucose control value is too high or too low, the process is said to be out of control and action is taken.

Now,  Mr. Arthur is trying to push SPC software not for a process but for errors in the process. For example, he uses the infection rate in a hospital. But the infection rate error is not a process that one wants to control – of course one does not want it to become worse - but its target is zero.

A more useful example than the hypothetical one provided by Mr. Arthur was published recently (1). Here, the authors were faced with an undesirable hospital infection error rate and set out to observe where errors occurred in the process of placing central lines. They then provided control measures and continued to track the error rate, which was reduced to zero. This is not SPC! It is much more like a FRACAS (Failure Reporting And Corrective Action System).

In another part of the article, Mr. Arthur suggests that “never events” can be tracked by SPC. Never events – a list of 28 such events have been put forth by the National Quality Forum – have as implied, targets of zero. Such an event is wrong site surgery. One should use something like FMEA (Failure Mode Effects Analysis) to reduce the risk of such events. It is silly to suggest SPC software for never events.

References

1.       An Intervention to Decrease Catheter-Related Bloodstream Infections in the ICU. Pronovost P, Needham D, Berenholtz S, Sinopoli D, Chu H, Cosgrove S, Sexton B, Hyzy R, Welsh R, Roth G, Bander J, Kepros J, Goeschel C N Engl J Med 355:2725, December 28, 2006

6/5/08 - Westgard Quality Control Workshop – Part 3

I just returned from the Westgard quality Control Workshop, where I was a speaker and have a few blogs worth of comments – this is the third.

EQC – Equivalent Quality Control

This is the CMS proposal (1) to allow clinical laboratories to reduce the frequency of quality control from twice per day to once a month given that 10 days of running QC shows no values that are out (and given some other conditions).

Let’s try to construct a hypothesis to base such a recommendation. For example:

given any possible error condition that could be detected by external quality control, internal quality control would detect the same error 100% of the time.

This is about the best I can think of, which would result in the recommendation:

Stop running external quality control.

What does running 10 days of external QC with no out of control results show? The answer is nothing. This is because one can assume that during these 10 days, there were either no errors or if there were errors, external QC was not able to detect them. (It is possible that internal QC detected errors during these 10 days). In fact, this experiment is guaranteed to be meaningless. To see this, one must realize that internal QC is always “on” and precedes external QC. So to see if external QC is redundant to internal QC for an error, would mean that internal QC would detect the error and either shut down the system or prevent the result – this being the external QC sample – from being reported. However, one can get different information by running external QC for a longer period because if internal QC misses an error but external QC detects the error, then one has proved that external QC is not redundant to internal QC. This was shown to me (2) as out of control results for a range of assays ranging from 1 to 10 per year, where these were real problems. Since controls are run twice per day, the number of affected patients samples is larger.

So a lab that reduces external QC to once a month is risking an even larger number of bad patient results which is made worse since the clinician has probably acted on the erroneous results.

Rather than do the experiment suggested by CMS, a lab can simply examine its external QC records for a sufficient length of time.

References

1.       To review, see: See http://www.aacc.org/events/expert_access/2005/eqc/Pages/default.aspx

2.       Personal communication from Greg Miller of Virginia Commonwealth University

6/5/08 - Westgard Quality Control Workshop – Part 2

I just returned from the Westgard quality Control Workshop, where I was a speaker and have a few blogs worth of comments – this is the second.

How does one determine acceptable risk

This was one of the questions asked by a participant – are there any guidelines? I also commented recently, that in spite of all of talk about risk management and putting in place control measures until one has acceptable risk, no one knows what acceptable risk means. Here’s some more thoughts on this.

There are different risks (1). These can be enumerated. These include:

perception – complaints from either hospital or non hospital staff

performance – traditional quality, including errors that can affect patient safety

financial – errors that threaten the financial health of the service including lawsuits

regulatory – errors that threaten the accreditation status of the service

So first, one must say which risk one has in mind. One can envision an acceptable regulatory risk (we always pass inspections) but an unacceptable patient safety risk.  Note also, that the risks are not necessarily unique. One can have a patient safety failure with or without a lawsuit.

Assume the risk in question is the performance risk and specifically about patient safety. The Cadillac version of assessing risk would be to perform a quantitative fault tree and arrive at a numerical probability of patient risk. This is unlikely and one would probably have a qualitative assessment. Whether the assessment is quantitative or qualitative, this still hasn’t answered the acceptability question.

The problem is there is no easy answer to this question. If one had unlimited funds, one could lower the risk to whatever level was desired but funds are limited by the economic healthcare policy of the laboratory’s country (2). So one answer of acceptable risk is how this economic policy is translated into regulations. (e.g., one follows existing regulations and passes inspections). Yet, this is only a quasi legal way of stating acceptable risk.

Recommendation

I suggest that risk be assessed by traditional means (FMEA, fault tree) which includes a Pareto chart or table to rank the risks. Then, if one optimizes the money that one has in implementing control measures (mitigations) by a portfolio type means, then one has an acceptable risk under the imposed financial constraints.

References

1.       Managing risk in hospitals using integrated Fault Trees / FMECAs. Jan S. Krouwer, AACC Press, Washington DC, 2004.

2.       See http://covertrationingblog.com/

6/5/08 - Westgard Quality Control Workshop – Part 1

I just returned from the Westgard quality Control Workshop, where I was a speaker and have a few blogs worth of comments – this is the first.

What’s Missing from Clinical Laboratory Inspections

At the Westgard Workshop, most of the participants were from clinical laboratories and I was impressed with how smart these people are. I also got a sense of a tremendous regulatory burden. From the CAP CD, I obtained at the Workshop:

      The mission statement of the CAP Laboratory Accreditation Program is:

“The CAP Laboratory Accreditation Program improves patient safety by advancing the quality of pathology and laboratory services through education and standard setting, and ensuring laboratories meet or exceed regulatory requirements.”

I have had mixed feelings about inspections that certify quality and have previously reported my experience with an industry quality program – ISO 9001 (1).

Here’s my assessment of clinical laboratory inspections to certify laboratories. It would seem that the premise of these inspections is to ensure that specific policies and procedures are in place and executed as proven largely by documentation, which guarantees high quality. So what’s missing? As far as I can tell – and it is with great difficult to read through these materials – that there is no measurement of error rates. Without such measurements, quality is unknown.

Recommendation

The regulatory bodies would describe a list of errors and their associated severities. The severities would be given numerical values such as the VA hospital system which uses 1-4. Every clinical laboratory would record each error (failure mode) that occurs in their laboratory, its severity, and its frequency (default frequency is of course 1).  They would multiply frequency x severity for each unique error (failure mode), add this up and get a rate by dividing by the number of tests reported per year.

Failing to count errors would be a serious violation.

This would be the start of a new premise for the regulatory bodies. Measure quality – if it’s unacceptable, the clinical laboratory would suggest and implement process changes. It’s a simple closed loop process. With emphasis on measurement, reliance on documentation should decrease and inspections should be less burdensome.

References

1.       Krouwer JS. ISO 9001 has had no effect on quality in the in-vitro medical diagnostics industry. Accred. Qual. Assur. 2004;9:39-43

5/4/08 - Acceptable Risk – Easy to talk about, but no one knows what it means

Standards about risk management always talk about “acceptable risk.” This is a qualitative term. Unfortunately, for much of healthcare there is no matching quantitative assessment or goal. Consider two examples.

It is possible to estimate the probability of a severe adverse event and to have an associated goal for such a probability but no one in healthcare does this. So one will see things like, “with this mitigation we have reduced the risk of the adverse event to an acceptable level” but the reality is no one knows what this really means.

5/3/08 - Never Events – Never a meaningful goal

This has been considerable discussion about the National Quality Forum’s  so called 28 never events (1). Here are some problems with this concept.

Never is a poor goal – Adverse events can be considered within a risk management program. Risk is the combination of two items – severity and probability of occurrence. By their selection, one can gather than severity is high for the 28 events. However, probability can never be zero. Consider a simple example. The likelihood of performing wrong site surgery is X. One performs a double check to prevent wrong site surgery. Now the likelihood is 0.0001X. But the double check can fail. So one can perform a triple check. Now the probability is much lower but it is still not zero. And so on. Working with probabilities (as in fault trees), is one way to see that probabilities are never zero, nor is risk.

28 goals are too many – If one wants to manage anything, one needs a limited number of goals. There is no reason why one can’t combine events to give a single goal – the overall risk of an adverse event.

“largely preventable” is not the same as preventable – In the NQF site, the never events are said to be largely preventable. The problems with this are obvious.

References

1.       See http://216.122.138.39/projects/completed/sre/index.asp

3/19/08 Alternatives to Six Sigma

This entry continues where the entry (Six Sigma can be dangerous to your health) left off. Given the problems with six sigma, what are some solutions to estimate the quality of an assay, using hCG as an example assay.

First, when total analytical error is calculated to estimate the values in zones A-C in an error grid, one should use conservative methods such as the empirical distributions suggested by the CLSI EP21A method, and where no data are deleted. Let’s say a clinical laboratory has done this evaluation with 40 patient samples for a new and reference method and found no results in zone C for an hCG assay. What can one conclude? Although there are 0% of the values in zone C, the 95% confidence interval extends to 7.2%. This means that for every million hCG results performed, up to 72,000 results could be in zone C. This is not very comforting and these types of evaluations don’t prove much, although one knows that the 7.2% rate is unlikely (because if this rate to occurred, it would be noticed).

FMEA is an approach that will provide an answer to the quality question but in its complete form, it requires considerable effort. To complete a FMEA analysis, one has to postulate all possible reasons why a result could fall into zone C. To get an idea of what is involved, take two possible failure modes, HAMA interference and a patient sample mix-up.

HAMA interference – To estimate the likelihood of a zone C result from HAMA interference, one needs to know the level of HAMA that will cause erroneous results in the assay and the probability of such levels in the population being sampled. Contacting the manufacturer might give one the level of HAMA to watch out for – I am not familiar with data about the distribution of HAMA in patient samples. Yet, one knows HAMA interference occurs (Clinical Chemistry. 2001;47:1332-1333).  

Patient sample mix-up – There are some data for patient sample mix-ups (Archives of Pathology and Laboratory Medicine: Vol. 130, No. 11, pp. 1662–1668). However, it seems that these cases are caught within the laboratory. One would need to determine how many cases actually are not caught within the laboratory. One could then model the likelihood of a zone C result by sampling from the empirical distribution of hCG results that are observed on the lab to see the likelihood of a mix-up causing a zone C result.

Because there are so many existing data in a clinical laboratory, one may also have the opportunity to perform FRACAS types of analyses. That is, in addition to modeling probabilities, once could use existing data to count actual failures.

One must then continue:

• with each other possible failure mode, calculate the probability of zone C results
• calculate the overall probability of zone C results (from all failure modes) and determine if that risk is acceptable
  • special software is typically used to perform these calculations
• construct a Pareto table if the overall probability of zone C results is too high and
• propose control measures to lower the overall risk to an acceptable level
  • the control measures must of course be affordable

At this point, one can get the idea that this level of effort is out of reach for clinical laboratories since the level of expertise and work need just to estimate the likelihood of a zone C result is huge. Even if a clinical laboratory could perform this task, it makes no sense to require every clinical laboratory to do so.

One possibility is to have a standards group tackle such a task., although this too has limitations as was shown for a (universal) control measure to prevent wrong site surgery.

Another possibility is to perhaps leverage resources beyond the clinical laboratory. For example, one could insist that before treatment for trophoblastic carcinoma, an hCG result should be confirmed either by performing a reference assay or perhaps by treating the sample and rerunning it. This requires an interaction between the clinical laboratory and clinicians.

So there are no easy answers to preventing severe, low frequency failures, (that cause patient harm) but as discussed before, coming up with a sigma estimate for an hCG assay, is also not the answer. Nor is doing nothing.

3/15/08 - Jan gets an award

I recently spoke at the Quality in the Spotlight conference in Antwerp, Belgium and gratefully acknowledge being awarded the Westgard Quality Award. This award was presented by Jim Westgard himself. The Quality in the Spotlight conference is a two day conference in Antwerp, devoted each year to a quality theme. This year’s theme was quality tools. I spoke about FMEA on each of the two days. It wasn’t until the second day of the conference that I realized that some of the other presentations were bothering me – perhaps I had a case of brain jetlag. This is an interactive conference so had I been quicker I would have presented my concerns to the speakers. But this did not happen so my concerns are in the previous entry to this blog. Prof. Dr. Jean-Claude Libeer, who founded the conference and also spoke about me with respect to the award, said that it was my blog which impressed people. So perhaps my previous entry could be taken as an acceptance speech.

On the second day, per instructions, I attempted to do a “workshop”. This is in quotes because I had to involve the audience but was only given one hour. Had I to do this again, I would have given an award to one lady, who answered some of the questions I posed to the audience. One example – name a case of at risk behavior that you have experienced. Answer, a technician, who had trouble getting a barcode on a patient sample to register, scanned the barcode from another patient. So perhaps this is also an illustration of the need to perform a FMEA on a control measure (what can go wrong with implementing barcodes).

Another highlight of my trip was spending three days in Amsterdam and hearing that in spite of frequent mistakes, my Dutch is begrijpelijk (understandable).

3/13/08 - Six Sigma can be dangerous to your health

At a recent conference, there were several presentations about six sigma for clinical laboratory assays. To recall, sigma is calculated as Sigma = (TEa – bias)/CV where

TEa is the total allowable error
Bias is the inaccuracy of the measurement procedure
CV is the imprecision of the measurement procedure

The problem with six sigma is that’s it taken as a sole measure of quality – that is, if you have a high sigma value (greater than 6) then your assay is assured of high quality. The rest of this entry explains why this is wrong.

First, TEa (total allowable error) is often specially called out as medically acceptable limits. One need only read the ISO 15197 standard for glucose to see this connection. I have previously commented about this standard. The implied meaning of medically acceptable limits in shown in below.

This is simply not the real world. Taguchi long ago specified a more realistic quadratic model of worth, which is shown below, superimposed on the original figure but in green.

Thus points A and B are similar in bias and are similar in causing (or not causing) medically unacceptable results. It is also likely then that if point A is ok, then so is point B. It is only when one gets far away from these limits that one is almost certain to have results that can cause harm. This is shown below with point C.

This can also be expressed as an error grid such as those for glucose. So the “sigma” calculations really only express the zone A region (grey) where 95% or more of the results should be. Zone B (white) can contain up to 5% of the results and zone C (dark grey) should contain no results. The error grid contains more information since each set of limits is different for each concentration. An error grid is shown below, taken from FDA guidance. In the guidance, WM is the test method and CM is the reference method. (In the document WM=waiver method and CM=comparative method).

So the problem is that sigma only accounts for zone A, but patients are harmed by values in zone C!

Now one might argue that there is nevertheless a relationship between sigma and the three zones, meaning that high sigma values are unlikely to have values in zone C and low sigma values are likely to have such values. This is also not true. Here is why.

1.       Often incorrect models are used to asses total error – see here.

2.       In estimating bias and CV, outliers – the very values that cause harm - are often thrown out.

3.       All sigma calculations are based on the assumption that the data are normally distributed. Most data do not fulfill this criterion. This means that often there are more frequent values in the tails of the distribution (again, this is zone C) than expected by calculations based on the normal distribution

4.       And maybe the biggest reason of all, values can occur in zone C that have nothing to do with the analytical process. If there is a patient sample mix-up, this can occur and these values are excluded (when detected) from virtually all analytical evaluations.

Think of it this way. If a loved one suffered medical harm, due in part to an erroneous lab result, would it make you feel better to know that the assay had a high sigma value? And would you associate that assay with quality?

I will comment on how one can address these issues in a future entry.

3/3/08 - At risk behavior

I am involved in risk management standards for clinical laboratories, where the focus has been on understanding how manufacturer’s devices can fail and how a clinical laboratory can put in place control measures to prevent these failures from causing harm.

My concern with these standards is that there is not enough emphasis given to the clinical laboratories own sources of error – its people. Among problems related to human errors are cognitive errors, non cognitive errors, reckless behavior, and at risk behavior – the topic of this entry.

At risk behavior is behavior that increases risk where risk is not recognized, or is mistakenly believed to be justified. Anyone who manages people must have had the experience by hearing  (perhaps second hand) “I don’t think that’s necessary and I’m not going to do it.” And of course, parents are familiar with at risk behavior practiced by their children.

An example of healthcare at risk behavior is reusing syringes. This occurred recently at an endoscopy clinic in Nevada and has affected up to 40,000 people. In reading the patient empowerment blog, one learns about other cases of reused syringes. In a case in Long Island, the physician reused syringes only for the same patient, but the syringes were used with multi-dose vials and these vials were used across patients.

In the recent case of reducing central line infections, Dr. Peter Pronovost observed that of the steps associating with placing a central line, in a third of patients, doctors skipped at least one step. Whereas, some of this could be attributed to non cognitive errors (slips), it could also be associated with at risk behavior. The control measure that worked here, was a double check step, whereby another healthcare provider would check to make sure each step was followed.

Discovering at risk behavior may not be easy, hence it needs to be on one radar’s screen.

2/14/08 Should one focus on a failure in a procedure or the outcome of such a failure?

Withholding payment for adverse events is a financial incentive to promote patient safety. Whether this incentive makes financial sense is something I will comment on later or perhaps not at all. For now, my comments are about the policy as it recently appeared (1).

The authors suggest the following criteria to withhold payment.

·         Evidence demonstrates that the bulk of the adverse events in question can be prevented by widespread adoption of achievable practices.

·         The events can be measured accurately, in a way that is auditable.

·         The events resulted in clinically significant patient harm.

·         It is possible, through chart review, to differentiate the adverse events that began in the hospital from those that were “present on admission” (POA).

The problem is with the third bullet and can perhaps be illustrated by the following figure.

In this figure FMEA events are shown by the dashed line.  The red dashed line is before FMEA. The green dashed line shows that after a successful FMEA, risk of failures has been reduced. FRACAS events are shown by the solid lines. The green line shows a reduction in the failure rate after FRACAS.

Keep in mind, for the dashed lines (FMEA), no failures have occurred, while for the solid lines, failures have occurred.

Now the policy defines a failure as an adverse patient outcome. One can view outcomes as the end of  an event cascade as in the next figure.

Assume that event C is an adverse patient outcome. According to the policy, payment is withheld only when event C is observed. In the first figure, the relevant concern area is shown by the ellipse as it is assumed that these are all high severity (severe patient harm) events.

This policy therefore excludes the following cases:

All FMEA events. That is, a procedure with a correctable high risk will be excluded from this policy because the event has not yet occurred. Considered the case of the Duke transplant error (2), before it happened. One can infer that this was a high risk procedure that would have benefited from a FMEA. In essence, this policy waits for disasters to happen.

All near miss events. Consider the case of the patient who had an MRI (3). Blood pressure monitor tubing had to be disconnected for the MRI. After the procedure, the tubing was incorrectly connected to an IV line. Before air was delivered from the automated blood pressure monitor, a family member noticed that things didn’t look right and contacted a nurse, who corrected the problem. Thus, there was no adverse event.

All defective procedures that don’t result in severe patient harm. Consider a healthcare worker who violates hospital policy (at risk behavior according to Marx (4)), which results in a patient fall. In this case, the fall results in a minor injury.  This is an important case because the policy fails to properly reflect risk management principles.

For a procedure that has a problem (e.g., a failed event), one has to classify the severity of the failed event and its probability (FMEA) or frequency of occurrence (FRACAS). The severity is classified not necessarily by the failed event but by the effect of the failed event. The effect is itself an event and can be a spectrum of severities. In the case of a patient fall, there is a distribution of harm associated with the fall event – some falls will result in severe harm, some will result in minor harm. Traditionally, in risk management, if severe harm is possible, then severity is associated with severe harm, even if the probability of severe harm is low. In this sense, severity is equated with potential outcome, regardless of whether that specific outcome has occurred.

One also has to classify the probability (FMEA) or of frequency of occurrence the event (FRACAS). Here, assume FMEA, one could choose between the probability of the failed event or the probability of the effect of the event (the adverse outcome). It is recommended to use the probability of the failed event, not the probability of the effect of the event. This is because one usually has control over the failed event and does not have control over the effect of the event.

Example: If a clinical laboratory provides a clinician with an erroneous result and the effect of that could be patient harm, the event is classified as severe. The probability is the probability of erroneous result, not the probability of patient harm, because patient harm is outside of control of the clinical laboratory (the clinician might not act on the result, might suspect it is erroneous and request it to be repeated, and so on).

Summary

References

1. Wachter RM ,Foster NE and Dudley RA Medicare’s Decision to Withhold Payment for Hospital Errors: The Devil Is in the Details The Joint Commission Journal on Quality and Patient Safety 2008;34: 116-123, see http://psnet.ahrq.gov/resource.aspx?resourceID=6760
2. See http://www.cbsnews.com/stories/2003/03/16/60minutes/main544162.shtml
3. See http://www.ismp.org/newsletters/acutecare/articles/20030612.asp
4. Marx, D. Patient Safety and the “Just Culture”: A Primer for Health Care Executives http://www.mers-tm.net/support/Marx_Primer.pdf

1/24/08 Software Verification and Validation

In spending two sessions with groups of people who verify and validate medical device software, I got the impression that most effort is spent on testing code (to the requirements that exist). In part, I based this assessment on the amount of questions (e.g., interest by the audience) when code testing was discussed vs. examining requirements. Yet, in reviewing recalls, and my experience in the IVD industry, I suspect that that most errors are caused by wrong requirements (see figure).

This makes me recall some definitions.

Bug – A coding error that prevents the software from meeting its stated requirement. A divide by zero error is a bug, but if the denominator can never be zero, this bug will never be a failure. Never be zero means the value can never be zero without a code logic statement such as If X <> 0, then … If the code logic statement were present, there would be no divide by zero bug.

Failure – Any deviation from customer expectations. This rather liberal statement is similar to the general definition of quality by ASQ. Each failure must be evaluated by the software / product development team to decide whether they agree and of course deviations have non software causes.

Example – A home glucose meter produces a value over 500 mg/dL. The meter displays ERR1. This is a requirement errors. It is known the value is too high ( it could be 501 or 1,000). The meter should say something like HIGH.

1/4/08 FMEA vs. FRACAS

I have previously compared FMEA and FRACAS, here. Another simple difference is:

(Successful) FMEA reduces risk.

(Successful) FRACAS reduces failure rates.

Now, one often hears about successful FMEAs. In my experience, these are not FMEAs, they are examples of FRACAS. An example is here. How can one tell that this is FRACAS and not FMEA. It’s simple - what is described is the reduction of a too high failure rate to a lower rate. With FMEA, the failure rate is zero – the event has not happened. What one does is to reduce the risk of this potential failure, from some amount to a lower amount. This is perhaps one of the reasons, one does not hear too much about FMEA successes. As I said before, to say that something that has never happened is now even less likely to happen (due to FMEA) just isn’t too exciting.

To reduce failure rates is a good thing and it is not a big deal to call this FMEA when it is FRACAS. However, it is simple to use the correct terms and if one doesn’t one might wind up neglecting to perform FMEA when it's needed.

1/1/2008 A Different Animal

I have spent my career in industry in R&D in a quality role. As I continue to interact with people that deal with quality in the in vitro diagnostics industry, I get the impression that most of these people are not from R&D but rather from regulatory affairs. What’s the difference? My perception is that regulatory affairs professionals focus more on compliance – I have focused on measuring things. Compliance is often assessed through audits with documentation a large part of audits. Measuring things forces activities to focus on improving the metric of interest. Documentation is of less importance.

What’s another difference? Whenever I write an article for publication on quality, it’s reviewed by regulatory affairs professionals. I can tell by the comments (e.g., they disagree with most of what I say). R&D people agree with me.

12/9/2007 - Frequency of QC in the clinical laboratory

Kent Dooley has written an interesting essay, which is here. One of the points he makes is that not all clinical laboratory errors result in patient harm because clinicians will not always act on the erroneous result. So if an assay result doesn’t agree with other clinical data, the clinician may suspect the result might be wrong and ask to have it repeated. Dooley suggests that the minimum QC frequency should follow the time course for the likelihood of a clinician requesting a repeat sample, so that upon repeat, if the result had been in error, the new result will be correct (because now QC has been run).

Now, I am unencumbered by the knowledge and experience of working in a lab but my view of things is somewhat different. It seems to me that there are several error/detection/recovery possibilities as shown in the figure below.

The problem of waiting for a clinician (of for that matter a patient) to question a result, before running QC is that it doesn’t take advantage of the purpose of QC, which is shown below.

That is, one runs the assay and at some time QC. If the QC is ok, then the results are released to the clinician. If not, one troubleshoots the assay including possibly rerunning patient samples. Using this scheme, QC frequency should not be determined by a retest time course but rather by the turn-around-time requirement for the assay.

Now if the clinician requests a the assay to be repeated, and QC had already been run, it is unlikely that running a second QC will detect anything. QC has limitations in its ability to detect error (see figure below). Random biases and random patient interferences will not be detected by QC.

This figure came from previous considerations about equivalent QC, which are here, and here.

Besides suspecting assay error, many assay results are repeated because a condition is being monitored. Delta checks are a type of QC that is performed on these samples to determine whether the difference between results is expected. Exactly how the clinical laboratory could act on the knowledge that the clinician suspects that something is wrong with the assay result is a topic for clinical laboratorians to answer.

12/07/2007 - Central lines and FRACAS

One hears of FRACAS success stories (like the one below) and FMEA failure stories (like the wrong blood type organs transplanted at Duke). A reason one doesn’t hear of FMEA success stories is that to say that something that has never happened is now even less likely to happen (due to FMEA) just isn’t too exciting. FMEA success stories are often not cases of FMEA, they are FRACAS, since rate improvements are discussed. FRACAS failures – we tried something, it didn’t work – are not very interesting.

A recent article in The New Yorker (1) provides an example of a FRACAS success story.

In the article, there is no mention of FRACAS but many of the steps were followed. The issue was a too frequent infection rate in central lines. It is important that one can measure this rate. One knows how many central lines are used, infections manifest themselves and their cause can be determined by culturing the lines. Some undercounting is possible but the rate seems fairly reliable.

The man behind the work, Dr. Peter Pronovost, first observed events for a month within the context of the process of placing central lines (e.g., process mapping). Errors in the process steps were identified. Since these steps were simple, such as washing hands, one could partly view these errors as non cognitive errors. This suggests a control measure such as a double check to prevent such “slips”. Actually, besides slips, there may have been some at-risk behavior (2). This is behavior that increases risk where risk is not recognized, or is mistakenly believed to be justified. The main control measure used was a checklist, with the addition of having nurses double check to see that the checklist steps were properly done. Then the rate was measured again and found to be considerably lower. All of this was published (3).

It was mentioned that an alternative control measure had been tried; namely, using central lines coated with antimicrobials. This expensive control measure failed to provide a substantial reduction in infection rates. This illustrates that one must be open minded when selecting control measures. There is sometimes a bias towards fixing the “system” (e.g., such as with coated lines) rather than fixing a people issue (e.g., which often implies blame). Dr. Pronovost implemented some system control measures by getting the manufacturer of central lines to include drapes and chlorhexidine – items that should have been available at the bedside but often were not.

Another big part of this story is ongoing resistance towards implementing this control measure more widely, even after it has been shown to be effective and low cost. Any control measure can be viewed as a standard and standards are not very popular. People will argue “but our situation is different”, “ICUs are too complicated for standards”, and so on. Financial incentives (or disincentives) for standards (e.g., P4P) loom. Dr. Gawande goes on to say how complicated things are in an ICU, yet there is precisely where standards helped. A similar situation happened in anesthesiology in the late 70s and early 80s. (Here, critical incident analysis was used and is basically the same as FRACAS.) The error rate was too high, effective control measures were developed, and widespread implementation of the control measures took considerable effort. You can read about that story here.

References

1.       Gawande A. Annals of Medicine. The checklist. The New Yorker, Dec. 7^th issue, 2007, see here (don’t know how long this link will work).

2.       Marx, D. Patient Safety and the “Just Culture”: A Primer for Health Care Executives http://www.mers-tm.net/support/Marx_Primer.pdf

3.       Pronovost P. et al. An Intervention to Decrease Catheter-Related Bloodstream Infections in the ICU. N Engl J Med 2006;355:2725-32.

11/28/2007 - ISO 14971 authors, expertise, and potential conflicts of interest

I have questioned the elevated status of ISO standards claimed by some. Often, people justify this status by asserting that ISO standards are prepared by a consensus of experts. This entry explores three topics related to this assertion:

·        ISO authorship

·        Expertise of authors

·        Potential conflicts of interest for authors

The membership of an ISO committee

If you have an ISO document – I have the latest version of ISO 14971 – one thing to notice is that there is no list of authors nor even a list of the committee members. I don’t understand why it is the policy of ISO to hide this information, nor could I find such an explanation (or list of members).

Note that CLSI (formerly NCCLS) has in each standard a list of authors and subcommittee members, advisors, and observers (as well as area committee members).

What does it take to be an expert?

A simple if not flip answer to this is to be on an ISO committee, since by assertion, all committee members are experts. Of course, for ISO committees, one cannot form an opinion, since membership is unknown outside of the committee.

Potential conflicts of interest

Here are some opinions about conflict of interest regarding ISO membership (given that I don’t have a clue who the authors are). To understand conflict of interest concerns, it is helpful to understand that ISO documents have quasi regulatory status. As such, organizations can be divided into two groups: regulatory providers, and regulatory consumers (see http://krouwerconsulting.com/Essays/StandardsGroups.htm)

Manufacturers – The membership from this (regulatory consumer) group is often filled with regulatory affairs professionals. Their potential conflict of interest is to shape the documents to favor ease of compliance. They favor horizontal over vertical documents (see http://krouwerconsulting.com/Essays/StandardsGroups.htm)

Clinical laboratory or hospital professionals – Although this group would not seem to have a vested interest, one can question, how many of these people serve as consultants for industry. If a standard is written for the clinical laboratory or elsewhere in the hospital than this group has the same regulatory consumer potential conflict of interest as the manufacturer.

Regulators – As a regulatory provider group, the potential conflict of interest is the healthcare economics policy in place by the current administration.

Consultants – This group often has a high potential conflict of interest since some consultants make their living by helping companies comply with ISO standards.

Trade associations – This group is the voice of manufacturers and if represented on a ISO group has the same potential conflict of interest as for manufacturers, but with the added concern that trade groups are skilled in organizing manufacturers.

Note that for CLSI, any prospective member must fill out a conflict of interest statement. I am unaware of anyone ever being turned away from membership due to the conflict of interest statements.

11/21/2007 - ISO 14971 and Residual Risk

The last entry was about FMEA goals, yet, the word “goal” isn’t in ISO 14971. Maybe “goal” suffered the same fate as the word “mitigation” – banned from ISO. There is an implied goal in ISO 14971 - the residual risk must be acceptable. To recall, residual risk is the risk that remains after control measures have been taken. Here’s where things get a little tricky.

In cases where the residual risk is unacceptable, one is supposed to perform a risk benefit analysis to determine if benefits of the medical procedure performed by the device outweigh any possible residual risk.

To frame this discussion, consider two types of residual risk:

1.       A residual risk from a known issue, such as an interference, where eliminating this risk is not “practical “

2.       The overall residual risk from unknown issues. A certain amount of effort is used to search for risks (e.g., through FMEA, FTA, and FRACAS). At some point, more effort is considered not practical. Note: One can look at FDA recalls to see that unknown risks are often found in released products and lead to recalls (1).

Use of the word practical in ISO 14971 implies that in some cases, risk reduction is too expensive. This is not meant to be pejorative since everyone has limited resources.

In most cases in the standard, the cost benefit analysis is positioned as an analysis of the medical device’s clinical benefit to the patient vs. its risk. But ISO 14971 does point out an additional frame for the discussion.

“Those involved in making risk/benefit judgments have a responsibility to understand and take into account the technical, clinical, regulatory, economic, sociological and political context of their risk management decisions.”

To understand the issue, consider Type 1 diabetes as an example with the medical procedure being use of a home glucose meter. Because of risks 1 and 2 above, the glucose meter will fail and provide an erroneous result, albeit rarely. This is the current status and it is clear the benefit of the home glucose meter outweighs the risk (e.g., ADA recommendations to test for glucose). Yet, if one conducts a thought experiment and starts raising the frequency of (all) home glucose meter failures, simple decision analysis (2) still warrants use of the device. That is, measuring glucose, even if it occasionally (e.g., more often than rarely) gives an erroneous result, is better (clinically) than not measuring it.

If a company is working on a home glucose meter which provided an erroneous result too often (e.g., compared to existing meters), they will keep developing the meter until its failure rate is competitive. That is, there is a hierarchy of requirements for release for sale and often the competitive requirements (features needed to sell the product – including quality) are more stringent than any medical need or regulatory requirement (3).

Would you pay 2.5 million dollars to go to Cleveland?

Richard Fogoros suggests that there is a limit that we can spend for healthcare (4). To make this point, he says that if a plane could be built that could be survivable for most crashes, most people would not pay for an astronomical ticket price.

So regulators could require lower failure rates (less risk), causing companies to invest more, which would result in higher healthcare prices, but this is not done because it is unaffordable, hence the level of risk allowed is usually driven by competition. This is risk management but it is not the clinical benefit risk analysis described in ISO 14971– it is financial risk management.

References

1.       See http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfRES/res.cfm

2.       Krouwer JS. Assay Development and Evaluation: A Manufacturer’s Perspective, AACC Press, Washington DC, 2002, Chapter 3.

3.       Krouwer JS. Assay Development and Evaluation: A Manufacturer’s Perspective, AACC Press, Washington DC, 2002, pp 38-39.

4.       Fogoros RN. Fixing American Healthcare. Publish or Perish Press, Pittsburgh, 2007.

11/17/2007 - FMEA goals in healthcare

FMEA is now a common risk management tool used in healthcare. Here’s a quick test. If the words “minimal cut set” and "Petri net" don’t mean anything to you, then you probably don’t have a quantitative FMEA goal. The rest of this entry explains some things to know about goals.

A quantitative goal must also be measureable and realistic. For example, a goal for imprecision (reproducibility) for a clinical laboratory sodium assay, might be 4% CV. One can measure this goal using a variety of experiments including those defined by standards such as the CLSI standard EP5A2.

FMEA deals with risk. Some common pitfalls about risk goals are:

·         A goal that an event should never happen. For example, the NQF (National Quality Forum) implies such by talking about “never events.” Risk is probabilistic and can never be zero. It is possible that an estimated risk is so low that in lay terms, it may be said to never be possible to occur but this lay usage is different from a formal quantitative assessment.

·         Too many goals. The NQF has a list of 28 “never events.” Virtually all of these cause serious patient harm. A goal could be restated in terms of patient harm, as the combination of risk from any of the 28 events.

·         The institute of Healthcare Improvement (IHI) implies goals in terms of evaluating the RPN (risk priority number) before and after implementing control measures. Some problems here are:

o   One may improve this metric by reducing the risk of less severe events (without reducing risk of severe events)

o   A severe risk with the lowest (categorical) probability of occurrence may be ignored as a candidate for improvement, since its RPN won’t change, but there still may be a way to lower risk (and still have the same (categorical) probability of occurrence rank.

Quantitative FMEA goals are possible and are used in the nuclear power industry although fault trees are used instead of FMEAs. Quantitative fault trees are evaluated among other ways using “minimal cut sets” and "Petri nets."

A reasonable non quantitative goal for FMEA is to learn more about potential failure modes. However, one should realize that it is difficult to assess how much is learned.

It is easy to have a quantitative FRACAS goal because it is easy to measure failure rates from observed failures, before and after implementing control measures.

11/10/2007 - Why FRACAS is important for medical device manufacturers

I have commented before that FMEA (and FTA) are used to prevent potential errors and that FRACAS is used to prevent the recurrence of observed errors. FRACAS is easier than FMEA, FTA because for FRACAS:

·         no modeling is required with respect to enumerating the possible failure modes (errors) – one simply observes the errors

·         one can easily calculate a failure rate, which can also help  predict when a failure rate goal will be achieved

From a user’s perspective (e.g., medical device customer), it is of course more important to prevent errors than to prevent their recurrence (e.g., no melt down vs. preventing another melt down). However, if FRACAS is completed before release for sale, then the FRACAS activity of preventing the recurrence of observed errors is also preventing potential errors from the user’s perspective, because (again, from the user’s perspective) the clock is at zero – no errors have occurred yet because the system hasn’t been used. This is summarized in the following table.

This does not mean that FMEA, FTA should be dropped. If a potential error has never been observed, one still must be sure that adequate control measures are in place.

So FRACAS is part of risk management in spite of the fact that it is not mentioned in ISO 14971.

Terms

FMEA – Failure mode Effects Analysis
FTA – Fault Tree Analysis
FRACAS – Failure Reporting And Corrective Action System
Failure Mode - Error

11/6/2007 - Some ISO 14971 risk control measures won’t reduce risk

The previous entry dealt with some limitations of the ISO risk management standard for medical devices – ISO 14971. This entry covers one of the limitations in more detail.

ISO 14971 fails to embrace the error – detection – recovery scheme, since they omit recovery. To see the problem, consider a clinical laboratory example in which a serum sample is analyzed for potassium.

Error – As the specimen is processed, some error occurs (OK, I am not that good at making up errors), which hemolyzes the specimen. If the cause of the error is known, then steps might be taken to minimize or eliminate it.

Detection – A technician visually examines the specimen before it is analyzed. The hemolyzed specimen is detected.

Recovery – The technician does not analyze the specimen and notifies the appropriate party to get another specimen. The end result depends on the turn-around-time requirement after re-assay.

If the turn-around-time requirement is met, no effect of the original error is observed

If the turn-around-time requirement is not met, the effect of the original error is a delayed result.

In either of the above cases, the error – detection – recovery scheme has prevented an erroneous result as the effect of the original error. (OK, one could get an erroneous result in the new specimen).

Whereas recovery in this case seems trivial, what if just as the technician is ready to perform the recovery, he/she gets called away and never performs the recovery. There is a well known example of a failed recovery where the error was the incorrect leg was scheduled to be amputated – the error was detected – but the recovery failed.  Although, the correct leg was identified in the operating room schedule (successful detection), there were multiple operating rooms and not all schedules were corrected (failed recovery) (1).

Where recovery becomes even more of an issue is when detection and recovery are located in different organizations. This is actually a common occurrence. For example, manufacturers detect a problem (this could be an official recall) and it is up to the hospital or clinical laboratory to follow the manufacturer’s recommendation as to the recovery (e.g., discard that lot of reagent).

In the risk management standard ISO 14971, a recommended control measure presents the opportunity for a failed recovery. ISO 14971 provides a hierarchy of risk control measures (mitigations), which in order of preference are:

1.       Eliminate the error

2.       Detect the error

3.       Inform the user of the error possibility (e.g., state a limitation of the procedure)

Number 3 is really part of detection (e.g., the detection is communicated). Number 3 is also commonly used for interfering substances for in-vitro diagnostic assays. This error is the stepchild for diagnostic assays. For example, I once surveyed a year’s worth of Clinical Chemistry assay performance complaints and found that interferences were the main complaint (2). One can speculate how this happened. A clinician realized that some treatment or patient status was inconsistent with a laboratory result, the laboratory investigated, and the assay result was found to be incorrect with an interfering substance as the cause of the erroneous result.

So consider the risk control measure for an assay whereby the manufacturer lists 10 substances that may interfere with the assay. How can the clinical laboratory “recover” using this knowledge (e.g., detection)? They can’t. To determine the concentration level of ten substances in every specimen is impractical (too expensive). So to review this situation:

1.       Eliminate the error – the manufacturer has tried, but failed. Ten substances still interfere (at or above certain concentrations)

2.       Detect the error – the only “detection” possible is to inform the clinical laboratory. Note that all other common detection methods (external quality control, internal algorithms) fail.

3.       Recovery – The clinical laboratory cannot perform a recovery

One should realize that whereas this is an undesirable state, it may be the best possible way of doings things given the economic constraints. As stated in the previous entry, the manufacturer is doing the right thing (as are regulators and the clinical laboratory).

However, the problem is that ISO 14971 would have us believe, that all risk is now at an acceptable level, which is not the case. The erroneous result is likely to occur, after which a cause is likely to be found since the manufacturer has stated a list of possible interfering substances.

Also, as in the previous entry, patient awareness is needed to be added to the mix as a significant way to prevent patient harm.

References

1.       Scott D. Preventing medical mistakes. RN 2000;63:60-64.

2.       Krouwer JS. Estimating Total Analytical Error and Its Sources: Techniques to Improve Method Evaluation. Arch Pathol Lab Med 1992;116:726-731.

11/4/2007 - Improvement is needed for risk management guidance for in vitro medical devices

When either a manufacturer or a clinical laboratory performs risk management, it is implied in the risk management standard ISO 14971 (and other literature) that risk management (1-4):

·         Identifies any product component or process step that has unacceptable risk

·         Through mitigations, reduces all remaining risk to an acceptable level

The purpose of this entry is to show that this doesn’t always happen and to suggest what to do about it.

Note 1: in order to understand ISO 14971, you need to learn ISO speak (“globally harmonized terminology”). For example, there are no lab “test results” or “assay results” - these are called “examination results.”

Note 2: ISO 14971 is intended for manufacturers. The section about risk management for clinical laboratories is based on my discussions with clinical laboratory directors.

The problem frame – ISO 14971 has a figure (H.1, page 61), which shows that there are three possibilities to prevent harm to the patient – the medical device manufacturer, the clinical laboratory, and the physician. ISO 14971 describes a mitigation* as either a way to prevent or detect an error. ISO fails to include recovery (5), which is a serious omission.

* I use here the word “mitigation” but should point out that mitigation has been banned from ISO speak and isn’t in ISO 14971.

An example problem– hCG (human chorionic gonadotropin) is an assay used to test for pregnancy. Such assays are subject to interferences, with HAMA (human anti-mouse antibody) a common example. In one case, a woman with an elevated hCG was diagnosed as having cancer and underwent chemotherapy, hysterectomy, and partial removal of one lung (6). Eventually, it was determined that she did not have cancer and all of the hCG assay results were incorrect due to HAMA interference – her actual hCG was not elevated. Cole studied this problem and found that it has occurred multiple times (7).

Manufacturer – One of the most difficult problems for manufacturers to overcome is lack of analytical specificity. This means that for many assays, a few results will be way off due to substances in the specimen that interfere with the assay. The fact that the rate of occurrence of this error is low is good, but as seen above, the consequences can result in severe harm to the patient. It is standard practice for manufacturers to accept the small rate of erroneous results and deal with the issue by stating these limitations in the product labeling (the package insert).

ISO 14971 provides the use of stating limitations as one method – albeit the least desirable method  - of risk reduction (H.4.1.c p70).

In the case of HAMA and other interferences, this warning is of little value to the laboratory since a laboratory has no information as to which specimens have HAMA or other interferences and it would be prohibitively expensive to try to determine this information (e.g., the recovery will fail). (I once had roof rack straps for my car which had a warning on the package – “stop every 25 miles to make sure the straps are secure”).

Clinical Laboratory – It was a surprise to me to learn from some clinical laboratory directors that:

·         They know that occasional erroneous hCG results are reported to clinicians, which ultimately causes patient harm

·         There is a quality control possibility to test a specimen for HAMA interferences by diluting it and rerunning it, but this is rejected as too expensive

·         Thus, clinical laboratory directors recognize the risk as unacceptable, but live with it

Analysis – The manufacturer is doing the right thing. If they could economically develop an assay without interferences, they would. Regulators who approve the assay are doing the right thing. Rejecting the assay would cause more harm to patients due to the lack of information of no assay result than the harm caused by a small number of erroneous results. The clinical laboratory directors are doing the right thing. If they reran too many samples, their costs would be too high and the laboratory would go out of business (more likely the laboratory director would be fired first and the rerunning process stopped).

The manufacturer notification of limitations, while necessary and conforming to ISO 14971, is ineffective to prevent risk. The clinical laboratory either does nothing to prevent risk or could potentially do the same thing as the manufacturer – issue a warning about potential interferences in the assay report to physicians.

Proposed Solutions – Recognize the problem. The current status quo of the risk management scheme is that after risk management has been performed there is no issue, which is wrong. Issuing limitations that are ineffective in reducing risk must be so acknowledged. The outcome of this risk management task for either the manufacturer or the clinical laboratory must result in the HAMA event as an undesirable* risk. It should be acknowledged that it is a work in progress to come up with a method – which must be economical – which reduces this risk to an acceptable level.

*Use of the term unacceptable risk makes no sense, since no one would tolerate unacceptable risk. Hence, a risk management program could through mitigations reduce previously unacceptable risk events to some combination of acceptable risk events and undesirable risk events.

The role of the physician and patient – I will leave the role of the physician to someone else. I suggest that the ISO figure above is wrong. It should have one more cascade; namely, the possibility for the patient to detect and recover from a problem and if this fails, then harm will occur. One should not discount patients as being not knowledgeable enough.  Through the use of the Internet, there is a growing movement for patients to take more control of their health. This includes assessing laboratory results which are playing an increasing role in medical decision making (for one example see reference 8).  So as part of a risk management program, one should include the patient.

References

1.       ISO 14971 http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=38193

2.       Can’t afford to buy ISO 14971? Then read summaries in Ref. 2-4 http://www.devicelink.com/ivdt/archive/06/03/011.html

3.       http://www.devicelink.com/ivdt/archive/06/04/009.html

4.       http://www.devicelink.com/ivdt/archive/06/05/009.html

5.       See Figure 4 in Krouwer, JS. An Improved Failure Mode Effects Analysis for Hospitals. Archives of Pathology and Laboratory Medicine: Vol. 128, No. 6, pp. 663–667. See http://arpa.allenpress.com/pdfserv/10.1043%2F1543-2165(2004)128%3C663:AIFMEA%3E2.0.CO%3B2

6.       Sainato, D. How labs can minimize the risk of false positive results. Clin Lab News 2001;27:6-8.

7.       Cole, LA Rinne, KM Shahabi S.and Omrani A. False-Positive hCG Assay Results Leading to Unnecessary Surgery and Chemotherapy and Needless Occurrences of Diabetes and Coma. Clinical Chemistry. 1999;45:313-314.

8.       http://men.webmd.com/news/20030527/high-psa-level-check-again

11/1/2007 - Who made ISO king

I have been working on a CLSI (Clinical Laboratory and Standards Institute) standard on risk management.  A preliminary version is available. This version needs revision and is getting it. As part of this process, comments are received and addressed using a consensus process. Having seen a few of the comments, one of them bothers me, not so much about the issue raised but the justification supplied, which is that the CLSI document deviates from the ISO standard on risk management – 14971. So this blog entry questions whether ISO documents should be taken as gospel.

I have commented  before on a specific ISO document – 9001. The title of my article says it all – “ISO 9001 has had no effect on quality in the in vitro medical diagnostics industry.”

ISO 14971 states things without providing any justification. There is a bibliography at the end but no links from the text to the bibliography. The document is not peer reviewed, although it undergoes its own consensus process. One is basically supposed to take ISO 14971 as correct because it is “based on an international group of experts”. I put the preceding phrase in quotes because, anyone serving on an ISO committee is automatically conferred expert status (this is true for CLSI committees as well).

So perhaps it is not even iconoclastic to question an ISO document, and one should certainly not suppress an idea because it deviates from an ISO document.

10/28/2007 - FDA Classes

Bob had a comment about my previous FRACAS post, which reminds me of something. In his comment, he refers to FDA device classes and says that Class II devices do not require as much rigor. FDA classes can cause some confusion because there are two types of classes - device classes and recall classes.

Devices classes are: class I, class II, or class III. It is class III that requires the most data and can “present a potential, unreasonable risk of illness or injury.”

Recall classes are also class I, class II, or class III. It is class I that is the most dangerous type of incident and can “predictably could cause serious health problems or death.”

Can one get a class I recall for anything other than a class III device? I don’t know the answer to this question but to a company, it is somewhat besides the point. Recalls are expensive, regardless of what device class they belong to or what the FDA requires for data and are to be avoided (e.g., using tools such as FRACAS).

10/25/2007 Fixing American Healthcare – A Review

My review of this book is from the perspective of a healthcare consumer and also as consultant to the medical device industry – I have no expertise in healthcare economics. In fact, the topic itself was initially of no interest for me – I figure we’re all going to get screwed and so someone talking about net present values of capitation expenditures would be a real snoozer. However, in this day and age of blogs, I came across the Covert Rationing Blog and found myself repeatedly coming back to this blog. Dr. Fogoros, aka DrRich, has a clear and entertaining writing style and made this topic interesting on his blog, so I bought the book. I was not disappointed.

The organization of this book is well thought out. The first 50 or so pages (out of slightly over 300) function as a summary of much of the analysis, after which people can either abandon ship or read on. I found Dr. Fogoros’s GUTH – grand unification theory about healthcare - to be quite compelling and also easy to understand. GUTH divides healthcare in four quadrants, all four combinations of centralized vs. the individual, and low quality and high quality. In this summary part, there is description of an investor session from 2000 which Dr. Fogoros attended. Here, Jim Clark (founder of Netscape) discussed his then latest venture – WebMD. I could have benefitted from Dr. Fogoros’s insight as to why WebMD would fail in its original concept, as I was one of the naive investors (fortunately only dabbling in this one). Simplifying insurers’  transaction costs and procedures was Jim Clark’s pitch, but the insurers did not want this simplification as their goal was to take money in but make it as complicated as possible to pay out for claims.

In the rest of the book, Dr. Fogoros supplies more details. What is so compelling to me is that when Dr. Fogoros exposes the forces at play, everything falls into place. There are no evil people, just people doing what they do best within the rules of society. So a football player that smashes his opponent on the field is cheered – off the field, the same behavior would land him in jail. In this book, the relevant players are like football players making hits on the field – they are not portrayed as evil.

Some of the discussions that were of interest: everything about money, the whole idea of covert (vs. open) healthcare rationing, the principle (that America refuses to abandon) that there can be no limits to healthcare, the destruction of the doctor patient relationship, the history and way HMOs work, why eliminating fraud won’t solve the healthcare cost problem, randomized clinical trials.

Two major groups are discussed as trying to control healthcare – the “Gekkonians” –who believe that market forces will reduce cost and the “Wonkonians” – who believe regulation can lower cost, largely by decreasing fraud.

Dr Fogoros has an engaging writing style. It is as if he is telling us a story, subtle humor is present  but the book is not a joke-a-thon.  One example - to illustrate the importance of cost in solutions, he says that one could do a lot more to make a plane crash survivable, but would you pay 2.5 million dollars for a ticket to Cleveland. Dr. Fogoros relays a chilling account of his own run-in with regulators, an experience that would make most people think of retirement. Thankfully for us, one reaction of his was to become an expert in the topi