CONVERGENCE AMONG DATA SOURCES, RESPONSE BIAS, AND RELIABILITY AND VALIDITY OF A STRUCTURED JOB ANALYSIS QUESTIONNAIRE

Research examined both the Position Analysis Questionnaire (PAQ) and the possible analyst sources for gathering job analysis information. 25 state government jobs were examined using job incumbents, supervisors, job analysts, and a comparison group of college students. Reliability and validity (judge convergence and prediction of present pay levels) information was determined for each of the four analyst categories. Results indicate that there is little difference between analyst sources, including students, in terms of their ability to reliably analyze a job using the PAQ. Convergent validity results showed a high degree of agreement among all judge categories when summing item frequencies across all 25 jobs. The prediction of present pay levels was significant for all judge categories but was noticeably smaller than previously reported studies. This seemed to be primarily due to the restriction in salary range of the present study. An analysis of judge response bias showed that supervisors and incumbents rate all or most PAQ items higher than their analyst counterparts. These findings suggest that who furnishes responses to a job analysis inventory makes little practical difference. The exception is that the determination of pay levels and human requirements for test construction purposes should be viewed with caution when different analyst sources are utilized for different jobs.