PT design is a fundamental stage in ensuring that a proficiency testing scheme provides a technically valid assessment of participant laboratory performance. ISO 13528:2022 establishes the main statistical considerations that should be defined before analysing the results obtained from an interlaboratory comparison.
A proficiency test is not simply a matter of distributing a sample, collecting results and comparing them. For the evaluation to be useful, the statistical design should be aligned from the outset with the objectives of the scheme, the type of data obtained, the characteristics of the results, the expected number of participants and the way in which results will be reported.
Clause 5 of ISO 13528 addresses these aspects and provides guidance for establishing the statistical design of Proficiency Testing (PT) schemes.
PT Design According To ISO 13528
ISO 13528 establishes a fundamental principle: there is no single statistical design that is appropriate for every proficiency testing scheme.
The PT design should be developed taking into account the specific objectives of the scheme and the characteristics of the data that will be obtained.
The main factors to consider include:
- The objective of the proficiency testing scheme.
- The nature of the data, whether quantitative or qualitative.
- The applicable statistical assumptions.
- The nature of possible errors.
- The expected number of results.
- The procedure used to establish the assigned value.
- The criteria subsequently used to evaluate performance.
Therefore, two PT schemes may require different statistical designs even if both have the general purpose of comparing laboratory performance.
Objectives As The Basis Of PT Design
Before selecting any statistical procedure, it is necessary to define what the scheme is intended to evaluate.
ISO 13528 considers different scenarios.
For example, a PT scheme may compare each participant’s result with a predetermined reference value, using limits established before the round begins.
In this case, the design should establish:
- How the reference value will be obtained.
- How the limits will be defined.
- Which method will be used to evaluate the results.
In other schemes, the objective may be to compare each participant’s result with the combined results of the group in the same round.
In this case, the design should determine how the assigned value will be calculated from the available results and which criteria will subsequently be used to assess performance.
A scheme may also be designed so that both the assigned value and certain dispersion parameters are derived from participant results.
Another possibility is to compare each laboratory’s result with an assigned value while taking into account the measurement uncertainty reported by the participant.
Finally, ISO 13528 also considers schemes specifically designed to compare the performance of different measurement methods.
Each objective therefore requires specific statistical decisions.
The Type Of Data Influences The Statistical Design
Another important element in PT design is the nature of the data obtained.
ISO 13528 distinguishes between several types of results, including:
- Quantitative data.
- Nominal or categorical data.
- Ordinal data.
- Discrete quantitative data.
This classification is important because statistical techniques cannot be applied indiscriminately to every type of result.
A numerical concentration result, for example, has different properties from an identification expressed through categories.
Similarly, some methods produce quantitative values on a discrete or discontinuous scale, as can occur with certain procedures based on serial dilutions.
In some cases, these data can be analysed using techniques normally applied to continuous variables, but the PT provider should first confirm that this treatment is appropriate.
PT Design And The Statistical Distribution Of Results
The expected distribution of results is another aspect that should be considered during the planning of the scheme.
Many techniques commonly used in proficiency testing assume that results from competent participants will follow an approximately normal distribution or, at least, a unimodal and reasonably symmetric distribution after transformation where necessary.
At the same time, interlaboratory comparisons may contain results that are far from the main population because of errors or other circumstances.
However, ISO 13528 indicates that it should not simply be assumed that every set of results is suitable for the automatic application of these techniques.
Symmetry Of Results
The standard notes that it is usually not necessary to formally demonstrate that results follow a normal distribution.
However, it is important to verify approximate symmetry.
This assessment can begin with a visual review of the data.
If the expected distribution is not sufficiently symmetric, the provider should use methods capable of properly accounting for this characteristic.
Possible approaches include:
- Transforming the data to obtain an approximately symmetric distribution.
- Using estimation methods resistant to asymmetry.
- Applying methods based on an appropriate statistical distribution.
- Using procedures resistant to extreme values.
The statistical analysis should therefore be adapted to the expected behaviour of the results.
The Case Of Microbiological Counts
ISO 13528 includes an example that is particularly relevant to PT schemes related to microbiology.
Results derived from dilution-based methods, such as certain quantitative microbiological counts, may follow a log-normal distribution.
In these cases, a logarithmic transformation may be appropriate before performing the statistical analysis.
Likewise, when dealing with small numbers of particle counts, the results may follow a Poisson distribution.
These examples show why PT design should consider the nature of the method and the expected statistical behaviour of the data in advance.
Using a statistical technique simply because it is commonly applied in other schemes may lead to inappropriate interpretations when the characteristics of the results are different.
Statistical Assumptions Should Be Justified
The proficiency testing provider should state the basis of the statistical assumptions used and demonstrate that they are reasonable for the expected data.
This justification may be based on:
- Observed data.
- Results from previous rounds.
- Previous experience of the provider.
- Available technical information.
- Technical literature related to the test.
In this way, the choice of statistical method is based on technical criteria rather than simply on the software used to perform the calculations.
What Happens When There Are Few Participants?
The number of participants is another key factor in the statistical design.
ISO 13528 states that the provider should consider the minimum number of participants needed to meet the objectives of the scheme.
In addition, the design should specify which alternative procedure will be used if that minimum number is not achieved.
This is important because statistical methods that are appropriate for a large group of participants may not be suitable when only a limited number of results is available.
The main difficulties include:
- Reduced reliability of certain statistics derived from participant results.
- The possibility that the comparison group may not be sufficiently representative.
- Greater influence of individual results on calculated parameters.
There Is No Universal Minimum Number Of Participants
ISO 13528 does not establish a single minimum number of participants that applies to every PT scheme.
The required number depends on several factors:
- The statistical procedure selected.
- The strategy used to deal with extreme results.
- Participant experience.
- Provider experience.
- The matrix being analysed.
- The measurand.
- The methods used.
- The specific objective of the scheme.
It is also important to determine whether participant results will be used to establish the assigned value, the dispersion used for performance evaluation, or both.
Therefore, having a small number of participants does not necessarily invalidate a PT scheme, but it does require an appropriate statistical design.
The Reporting Format Is Also Part Of PT Design
The way in which participants report their results can have a direct impact on the subsequent analysis.
For this reason, ISO 13528 includes reporting format among the elements that should be defined during PT design.
As a general principle, laboratories should carry out measurements and report results in a manner similar to their routine work, unless there is a specific reason to establish different conditions.
At the same time, the provider should use a consistent reporting format and minimise the possibility of transcription errors.
For example, units that are familiar to most participants may be used, and controls may be introduced to help detect the use of inappropriate units.
Electronic reporting systems can also help reduce errors during data collection.
How Replicate Measurements Should Be Reported
When the scheme requires replicate measurements, ISO 13528 recommends that participants report all individual results.
Only requesting the final average may not be appropriate when the design intends to use the information provided by replicates.
Depending on the objective of the PT scheme, the provider may request:
- All individual values.
- The mean of the measurements.
- Another estimate of location.
- The uncertainty associated with the result.
Replicate measurements can be especially useful when the scheme aims to obtain information related to participant precision.
How To Handle “Less Than” Or “Greater Than” Results
In certain methods, routine reporting practices may use expressions such as:
< limit of quantification
or
> upper limit
These results are considered censored data.
If numerical values are subsequently required to calculate performance indicators, the provider should define in advance how these results will be treated.
ISO 13528 considers two general strategies:
- Using validated statistical procedures capable of handling censored data.
- Asking participants to report the measured numerical value as well.
However, requiring a numerical value outside the range normally reported by the laboratory may lead to an assessment that does not fully reflect its routine service.
In addition, when a large proportion of results is censored, some methods used to derive consensus statistics may be adversely affected.
For this reason, the treatment of these results should be defined during the design of the scheme and communicated to participants in advance.
The Importance Of Significant Digits
The number of significant digits to be reported is also part of the statistical design.
In general, rounding error should be small compared with the expected variability between participants.
Insufficient resolution may cause many laboratories to report exactly the same result.
This can make it more difficult to calculate reliable location or dispersion statistics.
In some situations, the provider may therefore specify a particular number of significant digits to ensure that the results are suitable for the planned analysis.
Where participants report using different numbers of significant digits, this should also be considered when calculating consensus statistics.
PT Design: Planning Before Analysis
One of the main ideas conveyed by Clause 5 of ISO 13528 is that the statistical analysis of a proficiency test begins long before the results are received.
The provider should define in advance:
- The objectives of the scheme.
- The expected type of results.
- The statistical assumptions.
- The general performance evaluation procedure.
- The required number of participants.
- Alternative approaches if that number is not achieved.
- The reporting format.
- The treatment of replicate measurements.
- The treatment of censored data.
- Units and significant digits.
Only once these elements have been established does it make sense to apply the corresponding statistical procedures.
Defining these decisions in advance helps reduce problems during analysis and ensures that the results obtained genuinely address the objective of the proficiency testing scheme.
Statistical Design As The Basis Of A Reliable PT Scheme
Proficiency testing makes it possible to evaluate participant performance through interlaboratory comparisons carried out under predetermined conditions.
However, for that comparison to be technically useful, the statistical design should be appropriate for both the objectives of the scheme and the characteristics of the data.
Clause 5 of ISO 13528 provides guidance for defining this strategy before analysing the results, considering aspects ranging from statistical distribution to the number of participants and the reporting format used for measurements.
At SAHPYPRO, the design and development of proficiency testing schemes is approached from a technical perspective aimed at providing reliable and useful interlaboratory comparisons for participants.
Because in a PT scheme, statistics should not begin when the results arrive: they should be part of the design from the start.