Confidence Intervals and Sample Size for the ICC in Two-Way ANOVA Models

: The reliability of measurement instruments is vital in fields like medicine and psychology, where these tools are often used for diagnostic purposes. In reliability studies where participants are assessed by the same set of raters, the data can be modeled using a two-way ANOVA, with the intraclass correlation coefficient (ICC) as a key metric. This paper focuses on the ICC for agreement, which is crucial when the measurement values themselves, rather than just their rank ordering, are of interest. However, selecting appropriate confidence interval methods and determining adequate sample sizes for the ICC for agreement remains challenging. This work advances the understanding of confidence interval methods for the ICC for agreement, provides practical tools, and offers recommendations for selecting confidence interval methods and sample size procedures for planning reliability studies. In particular, we provide a comprehensive review and simulation-based comparison of six classes of confidence interval methods for the ICC for agreement identified in the literature. Our analysis includes a method based on the F-distribution, previously omitted, which demonstrates the best statistical properties in some cases. Then, in conjunction with the best-performing methods, we evaluate three sample size determination procedures based on the expected width of confidence intervals that we identified in the literature. To address the lack of accessible tools, we further developed an interactive R Shiny app, freely available to researchers, to compute confidence intervals and sample sizes. The utility of these methods is illustrated by a study on fetal heart rates.