Extreme-percentile identification collapses fastest
The paper's central consequence measure is the probability that a person whose true score lies in the top p percentile is actually placed there by the observed score. For p equal to 0.05, this probability falls from 1 when reliability is 1 to 0.05 when reliability approaches 0, at which point the test carries no information about the latent trait.
The paper reads this probability as the power of a statistical test for top-percentile membership. Because power of at least 0.8 is normally required, the plotted curve implies reliability should be at least about 0.9 for the test to be useful in this classification task. That is a stricter bar than the familiar 0.7 rule of thumb, and it applies specifically to identifying people at the extremes.
Ranking pairs and average placement degrade more gently
A second measure asks how often two people are ordered correctly. If one person's true score exceeds the other's, the probability that the observed scores preserve that order is 1/2 + sin⁻¹(√R)/π. At zero reliability the probability is 1/2, equivalent to a coin flip, and at perfect reliability it is 1.
A third measure tracks average placement. With imperfect reliability, a person's expected percentile moves toward the midpoint of 1/2, and the paper derives the average deviation from true placement as C = 1/4 − tan⁻¹(√R)/π. A weighted version that emphasizes both ends of the scale gives C = 1/6 − tan⁻¹(√R/(2+R))/π. These cost curves are similar to each other, and both decline as reliability rises, but they describe different decisions than the extreme-percentile probability.
Agreement between two tests is a testable prediction
The paper also derives the probability that two tests with reliabilities R1 and R2 rank two random individuals the same way. Unlike the earlier cost functions, this quantity yields a prediction that can be checked against data, because it depends only on the two reliabilities and the assumption that each observed score is normally distributed around its latent value.
This distinction is worth keeping in view. The percentile and placement measures describe how much information a single score carries about a latent trait; the two-test agreement measure describes consistency between two instruments. They answer related but separate questions and should not be substituted for one another.
Omega is preferred but costly in small samples
The paper reviews why attention has shifted from Cronbach's alpha to McDonald's omega. Alpha underestimates reliability except when the tau-equivalent model holds, whereas omega is derived under the common-factor model and is generally regarded as the better measure. The catch is estimation: omega requires fitting a model with many parameters, and in small samples maximum-likelihood estimation becomes difficult.
The paper examines this estimation problem directly, including bias and standard errors of the maximum-likelihood omega estimator in simulations. It also constructs profile-likelihood confidence intervals, which the paper reports perform well in practice and accommodate missing values easily. For the happiness dataset, the 95% interval extends further toward lower reliability, meaning uncertainty about the exact reliability value would reduce the test's value further.
A modified alpha tracks omega without model fitting
To avoid the computational burden, the paper derives a modified version of Cronbach's alpha that adds a factor-loading variance term to correct alpha's downward bias. When the common-factor model holds, this modified alpha closely approximates omega while requiring no parameter estimation.
The simulation table makes the comparison concrete. With 100 cases, 4 items, and true reliability 0.9, omega's standard error is 0.0229, while modified alpha is 0.886 with a standard error of 0.042; plain alpha is 0.784 with a standard error of 0.091. At true reliability 0.7 with the same design, modified alpha is 0.692 versus omega's 0.7, and at 0.4 it is 0.420 versus 0.4. The pattern holds across the other simulated designs, with modified alpha closer to the true value than plain alpha and sampling variability comparable to omega's.
Real datasets show when one factor is not enough
Across seven real datasets, the paper reports Cronbach's alpha, modified alpha, omega with 95% confidence intervals, a variance-weighted reliability Ri, and a two-factor reliability R2. Modified alpha often lands close to omega: for the happiness survey it is 0.626 against omega's 0.618, and for the airline satisfaction data it is 0.787 against 0.725.
The goodness-of-fit results set a boundary on the single-factor model. When there are more than a few variables, the one-factor model cannot capture the covariance structure, and at least two factors are needed; the airline data, for example, gives a chi-squared of 4,176,354 on 14 degrees of freedom with a p-value of 0. The two-factor reliability is sometimes lower and sometimes higher than the one-factor value, so the choice of factor structure changes the reliability estimate itself.
Classification thresholds depend on the decision being made
The paper's practical message is that the consequences of low reliability can be stated more explicitly than a blanket 0.7 rule. For identifying people in the top 5%, the power argument points to reliability of at least about 0.9. For correctly ordering two individuals or classifying someone above versus below average, the arcsine formula shows a gentler decline, reaching 1/2 only at zero reliability.
The paper also notes that weighting item responses by inverse variance raises reliability where it can be applied, but cannot be used for examination scores because students are told all questions carry the same maximum mark. The choice between plain alpha, modified alpha, and omega therefore depends on sample size, factor structure, and whether item weighting is permissible. The most consequential judgment is not which reliability coefficient is largest, but whether the intended classification sits near an extreme, where the paper's evidence demands substantially higher reliability.
Scope conditions and the IRT extension
The derivations assume normally distributed latent scores and item errors under the common-factor model, and the cost functions are built for multi-item scales scored by summing items. The real-data fits show that the one-factor model often fails once several variables are present, which limits how far the single-factor reliability values transfer.
The paper identifies item response theory as a further direction, because IRT produces a reliability that varies with the true score rather than a single scale-level value. Extending the cost-based perspective to that setting would let the same classification questions be asked where reliability is not constant across the trait range.
Sources used in this answer
Psychometric Tests: Quantifying the Consequences of Low Reliability and Improving Reliability Estimation
The paper derives consequence-based measures showing that extreme-percentile classification requires reliability near 0.9 for 0.8 power, and proposes a modified Cronbach's alpha that approximates McDonald's omega without model fitting.