Simulated Power Study of ANCOVA vs. Repeated Measure Analyses for Two-Way Designs with one Repeated Measure

Abstract

Whether one should use an analysis of covariance or a form of difference score test (difference as an outcome or repeated measure) is not always clear. The literature on the topic focused for a while on Lord's paradox which lead to the conclusion that both analyses were equally valid when there is true random assignment. Yet, the issue of which analysis is best was little explored. In an attempt to create a unifying simulation framework that will allow for comparable results when exploring various data structure variations, I will tackle 5 such manipulations by exploring the impact of varying effect size and relationship between time points, violating the homogeneity of the regression slopes assumption, exploring the effect of large systematic baseline differences, the impact of data missing at random, as well as comparing the sample size requirements for a given test power. The programs provided, which allow for tens of millions of simulations to be run in a reasonable time frame (within a day) also puts to rest any ambiguity on the stability of the results. By analyzing Type I error rate and statistical power, I establish that ANCOVA respects the type-I error rate of alpha, and has more power than repeated measure analysis in most cases, but should be avoided when there is a baseline imbalance. Hence, in cases where ANCOVA is applicable, it is preferable to use it over other difference score tests.