Evaluating Item-Sort Task Methods: The Presentation of a New Statistical Significance Formula and Methodological Best Practices

AbstractPurposeAn item-sort task is a common method to reduce over-representative item lists during the scale-creation process. The current article delineates the limitations and misapplications of the accepted statistical significance formula for item-sort tasks and proposes a new statistical significance formula with greater utility across a wider range of item-sort tasks.DesignFirst, a simulation study compares the two formulas in an array of conditions that vary on sample size and number of assignment choices. Second, an empirical study compares the results of three separate item-sort tasks across the two formulas for statistical significance.FindingsIn the empirical study, the proposed formula produces more correct retention decisions than the existing formula across all three item-sort tasks. In the simulation study, the proposed formula is more appropriate than the existing formula under most conditions. The two formulas function identically in item-sort tasks with only two assignment choices.ImplicationsResearchers could obtain erroneous results when misapplying the existing item-sort task statistical significance formula to cases with more than two assignment choices. The proposed formula corrects this limitation, ultimately providing accurate results more often than the existing formula. Applying the proposed formula could help future research and practice throughout the scale development process.OriginalityDespite widespread use, few attempts have been made to improve scale-creation pretest methods, particularly item-sort tasks. The current study demonstrates that even conventional statistical methods are susceptible to misuse and misapplication, and future research could benefit from the reexamination of other common methods.