Circular Error Rate & Consistency Coefficient

These metrics apply only to a Pairwise Comparison test and give an indication of how consistent each participant's responses are. They can be found in the .csv file obtained when Exporting Analysis Data.

They can be used to detect outliers, that is, results from one participant that may affect the statistical significance of the overall test results. However, these metrics can also reflect how difficult the test is. Consequently, each individual participant’s metric values should be regarded with respect to the rest of the participant panel, and outliers should be defined by comparison with the panel’s average metric value and standard deviation.

A circular error is defined, for a triplet of sounds A, B, and C, as the case where the following 3 conditions are met (considering any possible permutations of A, B, and C):

  • A is preferred to B
  • B is preferred to C
  • C is preferred or equal to A

A circular error therefore shows an inconsistency in how pairs are assessed for 3 given sounds.

Given N sounds, the circular error rate CER is defined as the number of circular errors NCE divided by the number of possible sound triplets, Ntriplets :

CER values are between 0 (no circular error) and 1 (only circular errors, which is technically impossible if N is greater than 3). Low CER values therefore indicate rather consistent results.

The consistency coefficient is derived from the circular error rate CER and the number of sounds N, as follows:

Consistency coefficient values are between 0 (low consistency) and 1 (high consistency).