An information processing approach to judges' agreement and disagreement patterns when encoding verbal protocols.

Abstract

An attempt is made at answering the following questions: How can reliability be assessed? Is the final decision reached congruent with the reasons given to arrive at that decision? Are there any particular qualities in judges which may have an effect on the encoding procedure? Does the type of task (well-defined or ill-defined) have an effect on the encoding procedure? It was predicted that judges who are more skilled at categorizing protocols would use certain strategies that less skilled judges would not. The level of agreement on the final decisions was analyzed under several criteria. The marked difference indicates that the codes selected by the judges are not always coherent with the reasons invoqued or that the grid categories are ambiguous. (Abstract shortened by UMI.)