Assessing the performance of the approximate chi-square and Stout's T statistics with different test structures.

Abstract

The assessment of dimensionality underlying the responses to a set of test items is a fundamental issue in applying correct IRT models to examinee ability estimation and test result interpretation. Currently, three assessment methods have been shown to be particularly promising: the original Stout's T (Stout's T1), the refined Stout's T (Stout's T2) (Nandakumar, 1987, 1993) based on Stout's essential unidimensional assumption (Stout, 1987), and the Approximate chi2 (De Champlain & Gessaroli, 1992) derived from McDonald's nonlinear factor analysis based on the weak principle of local independence (McDonald, 1981). However, the three indices have only been tested under limited research conditions. The purpose of this study was to assess and compare the Type I error rates and power of the Approximate chi2, Stout's T1, and Stouts T2 in assessing dimensionality of a set of item responses. The variables used in the Type I error study were test length (L) (40 and 80 items), sample size (N) (500, 1,000, and 2,000), and item discrimination (a) (.7, 1.0, and .14). A 2 x 3 x 3 design was created. For each cell of the design, 100 replications were carried out. In the power study, in addition to the three variables used in the Type I error study, different test structure (two dimensional simple test structure and two dimensional complex test structure) and dimension correlation (r = .0, .4, .57, and .7) were used. For both simple and complex test structure, a dimension ratio of 3:1 was set. Similarly, 100 replications were carried out for each combination of the conditions. A total of 14,400 data sets were generated. A according to the results obtained in this study, each index possesses certain advantages and drawbacks. The Approximate chi2 had good Type I error of zero over all conditions and excellent power with two dimensional simple test structure. Stout's T1 and Stout's T2, on the other hand, had higher Type I error rates than the Approximate chi 2, ranging from zero to 12, excellent power with two dimensional simple test structure, and better power than the Approximate chi2 with two dimensional complex test structure. However, Stouts T1 and Stouts T2 have to be used with great caution given the unsatisfactory power shown with two dimensional complex test structure in many cases. (Abstract shortened by UMI.)