Uniformity of product performance claims
The issue
For any diagnostic assay supplied by a manufacturer, there are a series of product performance claims. They are often found in the package insert or instruction manual but are also contained in product advertising and in all are formally called product labeling. The issue is that different manufacturers state claims differently and often in a way that is unclear so that a laboratorian has a hard time understanding what the performance claim means and how to compare claims from different manufacturers.
As an example, consider the imprecision of an assay, which for the purpose of the example is shown as identical for manufacturer A and B in Figure 1, but with different claims.
Figure 1
Precision claim for an assay by two manufacturers
Manufacturer A claims the precision to be at location A. What he means is that a user who conducts the precision experiment will on average estimate precision at location A. This also means that 50% of the time, observed precision will be better than the claim, and the other 50% of the time observed precision will be worse than the claim.
Manufacturer B claims the precision to be at location B. What he means is that a user who conducts the precision experiment should get a precision estimate that is always better then the claim or else there is something wrong with the system.
There are different reasons why a manufacturer would want one claim over the other.
Claim A is often favored by the Marketing department because it is lower (more favorable) than the B claim.
Claim B is often favored by the Service department because there will be fewer false complaints. Recall that for claim A, 50% of the time customers will experience worse precision performance than the claim. Whether there is actually something wrong with their system requires a statistical test, which is a more complicated situation than that of claim B.
History of a proposed standard to deal with product performance claims
In 1986, NCCLS (now called CLSI) proposed a project to deal with the claims issue. The project was appropriately called Uniformity of Claims, and was proposed by Richard Passey, a laboratorian who was well respected within NCCLS and was the recipient of their highest award. Naturally, there was keen interest in this document by manufacturers and through many meetings, issues such as those in Figure 1 were laid out. Eventually, in 1996 the document was published. Dont try to buy this document from CLSI it is no longer available! (see below).
CLSI documents are initially published as P=proposed documents. After revision, they are published as A=approved documents. This document (EP11 P) was one of a number of P documents which lay dormant for a number of years in the Area Committee on Evaluation Protocols. When I became chairholder of the committee in 1999, one of my directives was to advance (or cancel) all P documents. Of the various P documents, EP11 P was one that I championed, and through a working committee had a proposed EP11 A document.
Although this document was again supported by most manufacturers, there were some powerful manufacturers who did not want this document advanced and wished to have the project canceled. NCCLS management tried to get me to cancel the project and gave as reasons among others:
- It was costing too much money
- It was no longer needed
- There was an ISO document that replaced it
None of these reasons were convincing since:
It was costing too much money This was never mentioned for other similar projects which cost the same or more money
It was no longer needed No evidence was ever provided for this.
There was an ISO document that replaced it This ISO document did not in fact replace EP11 and is covered in more detail.
As a consultant, suffice it to say I am aware that reasons given for not wanting to do something (resistance) often do not include the real reason.
The ISO version of Uniformity of Claims
There is a trend in ISO documents to split them up into separate documents. In this case, there are the following documents:
18113-1 - Part 1: Vocabulary and general requirements 10 pages
18113-2 - Part 2: In vitro diagnostic reagents for professional use 10 pages
18113-3 - Part 3: In vitro diagnostic instruments for professional use 8 pages
18113-4 - Part 4: In vitro diagnostic reagents for self-testing 10 pages
18113-5 - Part 5: In vitro diagnostic instruments for self-testing 8 pages
What does this document collection say with respect to performance claims (in 46 pages)? Basically it says describe the performance.
There is an essay discussing vertical and horizontal performance standards. EP 11 is a vertical performance standard. It does not tell you how to perform a claims experiment but it does require a format for how claims are presented, explains why, and provides examples. I dont believe that the 18113 series even reaches the level of a horizontal standard with respect to providing guidance on uniform performance claims.
What finally happened
I refused to cancel EP11. The Area Committee on Evaluation Protocols approved the EP11 A document but the NCCLS Board of Directors canceled the project so EP11 P is no longer available and EP11 A was never issued.