The PT initial review is a fundamental stage before calculating assigned values, standard deviations or performance scores. ISO 13528:2022 devotes Clause 6 specifically to checking that the items and results obtained in a Proficiency Testing round are suitable before the final statistical analysis is applied.
This step is particularly important because a data set may contain problems that do not arise from the participants’ actual performance: unstable items, insufficient homogeneity, differences between measurement methods, transcription errors, unexpected distributions or extreme results.
Analysing these data directly without a preliminary review can lead to incorrect conclusions.
For this reason, ISO 13528 sets out a sequence of checks that helps the provider understand what is happening in the data before using them to evaluate participant performance. In this area, the standard complements ISO/IEC 17043, providing more detailed statistical guidance for scheme design, item validation and the review of results in PT schemes.
What Is The PT Initial Review?
The initial review is the process by which the proficiency testing provider examines the distributed items and the results received before carrying out the final statistical evaluation.
It is not simply a matter of checking whether some values are very high or very low.
The review should answer several questions:
- Were the items sent to participants sufficiently homogeneous and stable?
- Are all participants actually measuring the same measurand?
- Are there systematic differences related to the methods used?
- Are there obvious transcription or unit errors?
- Does the distribution of results have the expected shape?
- Are there extreme results and, if so, how should they be treated?
Resolving these questions before calculating performance statistics helps prevent a problem unrelated to the participant from affecting their evaluation.
Homogeneity And Stability Of PT Items
The first check established by ISO 13528 concerns the material distributed to participants.
The provider must ensure that batches of proficiency test items are sufficiently homogeneous and stable for the purpose of the scheme.
This is essential because the variability observed between participants should arise mainly from their measurement processes, not from significant differences between the items themselves.
ISO 13528 describes different ways of obtaining this evidence: through experimental studies, by using previous experience with sufficiently similar materials, or by evaluating the results of the current round for behaviour consistent with homogeneity or stability problems.
The choice of measurand is also important. Where possible, all relevant properties should be assessed. However, the standard allows a subset to be used if it can be demonstrated that those properties are sufficiently sensitive to detect potential problems affecting the remaining properties.
For example, a property that is sensitive to precipitation, settling or changes during processing can provide particularly useful information about the behaviour of the batch.
At SHAPYPRO, we have already covered this topic in detail in our article on ISO 13528 and homogeneity and stability in proficiency testing schemes.
In the context of Clause 6, the key point is that this check acts as the first barrier before interpreting participant results.
Are All Measurement Methods Comparable?
Once the items have been checked, another important question arises: the method used by each participant.
In some PT schemes, all participants use sufficiently comparable procedures and the same assigned value can be used for the group.
But this is not always the case.
When participants are allowed to use different measurement methods, some procedures may produce systematically different results.
In these situations, a single assigned value may not be appropriate for all participants.
ISO 13528 states that the provider should consider this possibility both during scheme design and when reviewing the data from each round. In some cases, different assigned values may be used for different method groups.
This point is particularly important because an apparent separation between results does not necessarily indicate poor performance.
It may show that there are two populations of results associated with methods that respond differently.
Identifying this before the analysis avoids comparing participants under conditions that are not truly equivalent.
Obvious Blunders Are Not The Same As Outliers
One of the most important aspects of the initial review is distinguishing between an obvious blunder and a statistically extreme result.
ISO 13528 recommends removing from the data set, before applying robust procedures or outlier tests, results that can clearly be identified as blunders.
A typical example would be a participant reporting a result in the wrong unit.
Another example could be the accidental interchange of results corresponding to two different proficiency test items.
These values are not simply results that are far from the rest: there is an identifiable cause showing that the reported value does not correctly represent the intended result.
In some cases, it may be possible to correct them, but only where an approved policy and procedure allow this.
At this point, the standard introduces a particularly important precaution:
If there is any doubt as to whether a result is genuinely a blunder, it should be retained in the data set and taken forward to the subsequent review stages.
This distinction prevents data from being removed simply because they appear inconvenient or too far from the group.
Visual Review Should Be Performed Before Analysis
Graphs are not only a way of presenting final results.
During the PT initial review, ISO 13528 considers the visual review of data to be one of the first stages of analysis.
This review should be carried out by a person with sufficient technical and statistical expertise and has two main objectives:
- Confirm that the distribution approximately follows the expected behaviour.
- Identify anomalies or sources of variability that were not anticipated.
For example, finding two clearly differentiated groups may indicate a bimodal distribution.
And that bimodality may have different explanations:
- Two measurement methods producing different results.
- Contamination of certain proficiency test items.
- Instructions that have been interpreted differently by participants.
The standard indicates that these situations should be investigated before proceeding with the statistical evaluation.
Histograms And Kernel Density Plots
A histogram can provide an initial view of the distribution and help detect asymmetry, clustering or extreme results.
However, its appearance depends considerably on how the intervals are defined.
For this reason, ISO 13528 also highlights the usefulness of kernel density plots, which can make it easier to identify multimodality or lack of symmetry.
The objective is not to choose the most visually attractive graph for the report.
The objective is to use graphical representations that can reveal whether the statistical assumptions defined for the scheme actually appear reasonable.
Robust Statistical Methods: Analyse Without Automatically Removing Extreme Results
During the PT initial review, after visually examining the data, one of the most important concepts in ISO 13528 comes into play: robust statistics.
Robust methods make it possible to estimate the centre and dispersion of a distribution while reducing the influence of extreme results, without first having to identify and remove every value regarded as an outlier.
This is an important difference from classical approaches in which certain results are first removed and a conventional mean and standard deviation are then calculated.
ISO 13528 indicates that, in general, robust methods are preferable to procedures based on deleting results labelled as outliers.
The estimators and procedures described in the standard include the median, scaled median absolute deviation, normalised interquartile range and several robust methods presented in Annex C.
One of the best-known is Algorithm A, which uses an iterative procedure to limit the influence of extreme values and obtain robust estimates of location and dispersion.
It is particularly useful when most results belong to an approximately normal population and there is a relatively small proportion of discrepant results.
For data sets where a larger proportion of extreme values may occur, the standard provides other robust estimators.
The final choice of method is the responsibility of the PT provider and should be aligned with the design and objectives of the scheme.
When Should Outlier Tests Be Used?
ISO 13528 does not prohibit the use of specific tests to detect outliers.
They may be used to support the visual review or where an exclusion strategy is considered necessary.
However, several precautions apply.
The tests used must be appropriate for the characteristics of the data, and their statistical assumptions must apply sufficiently for the intended purpose.
This is particularly important because many outlier tests are based on the assumption that the data come from a normal distribution.
If an outlier rejection strategy is used, the provider must document the procedure, the confidence levels required for rejection and the limits placed on the proportion of data that may be rejected.
The provider must also be able to demonstrate that the resulting statistical estimates remain suitable for the purpose of the PT scheme.
The standard refers to procedures such as Grubbs’ test and, where replicate results are available, Cochran’s test.
But there is one particularly important detail:
Removing a result from the statistical calculation does not mean removing the evaluation of that participant.
If a result is removed from the set used to obtain certain statistics because it is considered an outlier, the participant’s performance should still be evaluated using the same criteria applied to all participants.
This distinction is fundamental.
A result may be unsuitable for estimating the general behaviour of the group while still being precisely the result that must be evaluated for that participant.
The Order Of The Review Matters
The PT initial review described in Clause 6 can be understood as a logical sequence.
First, the provider checks that the items are not the source of the problem.
Next, it considers whether the methods used by participants are sufficiently comparable.
Only results that can genuinely be considered obvious blunders are then identified.
The distribution of the data is then examined.
Only after these steps should the provider decide which statistical procedure is appropriate and how possible extreme results should be handled.
This order helps avoid a relatively easy mistake: starting by looking for outliers.
A result that is far from the group can have several causes.
It may reflect a genuine problem with the participant.
But it may also reveal a methodological difference, an unexpected distribution, a problem with the proficiency test item or even a reporting error.
Statistics should not decide what has happened before the provider has a reasonable understanding of what the data are showing.
From The Initial Review To The Assigned Value
The initial review in Clause 6 also prepares the ground for the subsequent clauses of ISO 13528.
Clause 7 addresses the determination of the assigned value and its uncertainty.
When the assigned value is derived from a consensus of participant results, the standard itself indicates that the review techniques in Clauses 6.2 to 6.6 should be applied to confirm that there is sufficient agreement before combining the results.
This shows that the initial review is not an accessory stage.
It determines whether the set of results is genuinely suitable for calculating statistics that will subsequently have a direct impact on participant evaluation.
Why This Review Matters For A PT Provider
A Proficiency Testing scheme must be able to distinguish between variability attributable to participant performance and variability generated by other causes.
If this distinction is not made correctly, there is a risk of attributing to a laboratory a problem that actually arises from the design of the exercise, the material or the way the results have been grouped.
This is why the initial review combines statistical knowledge with technical expertise.
A graph may reveal two different populations.
Statistics can identify that separation.
But determining whether it corresponds to two analytical methods, a preparation problem or another cause also requires an understanding of the test and the behaviour of the material.
ISO 13528 emphasises precisely this combination of expertise: statistical tools are necessary, but they do not replace technical interpretation.
The Initial Review Within SHAPYPRO PT Programs
At SHAPYPRO, the statistical treatment of a proficiency test forms part of a broader process that begins before performance scores are calculated.
Item validation, data review and the selection of an appropriate statistical procedure help ensure that the subsequent evaluation is based on technically consistent information.
The initial review is particularly useful for detecting situations in which the data should be investigated before determining the assigned value or calculating statistics such as z-scores, z’-scores or other performance indicators.
In this way, evaluation is not reduced to automatically applying a formula to every result received.
First, it is necessary to determine whether those results can and should be analysed together.
Conclusion
The PT initial review is one of the fundamental stages of the statistical treatment established in ISO 13528.
Before determining assigned values or evaluating participant performance, the provider should confirm that the items are suitable, review possible methodological differences, identify obvious blunders, examine the distribution of results and select an appropriate statistical strategy.
The standard provides robust tools for handling extreme results, but it also makes clear that a different result should not be removed automatically simply because it is different.
That is precisely the purpose of the initial review: to understand the data before evaluating them.
Because in Proficiency Testing, reliable statistical analysis does not begin with the calculation of a z-score.
It begins by checking that the data on which the calculation will be based make sense.