The stability of item parameter estimates in the context of a second language competency test.

Abstract

The purpose of this study was to examine the feasibility of using IRT models to equate test versions of an English second language (ESL) test through the use of subsets of linking items. The test was designed to measure global comprehension at the intermediate level, of listening and reading texts and general knowledge of vocabulary, grammar and structure as measured by a cloze text. The data were subsets of listening, reading and cloze items used on two or more occasions. The procedure used was to ascertain the stability of the item parameter estimates from correlations and other descriptive measures. The estimates were obtained using three estimation methods (LOGIST, BILOG, NoHarm) and three IRT models. In addition the unidimensionality of the data sets was examined using a method described by Gessaroli and De Champlain (1991). The results suggest that IRT models may not be suitable with test data such as those used in this study. Failure could not be consistently linked to too few candidates, issues of statistical dimensionality, too few items, or the difficulty of the material in which the target subsets were embedded. If an expanded study yields similar results, then the issue is one of validity, whether the definition of ESL global comprehension at the intermediate level is compatible with how an ability is defined under an IRT model.