How to reduce the number of rating scale items without predictability loss?

Rating scales are used to elicit data about qualitative entities (e.g., research collaboration). This study presents an innovative method for reducing the number of rating scale items without the predictability loss. The “area under the receiver operator curve method” (AUC ROC) is used. The presented method has reduced the number of rating scale items (variables) to 28.57% (from 21 to 6) making over 70% of collected data unnecessary. Results have been verified by two methods of analysis: Graded Response Model (GRM) and Confirmatory Factor Analysis (CFA). GRM revealed that the new method differentiates observations of high and middle scores. CFA proved that the reliability of the rating scale has not deteriorated by the scale item reduction. Both statistical analysis evidenced usefulness of the AUC ROC reduction method.

[1]  Ricardo Araya,et al.  School-based intervention to improve the mental health of low-income, secondary school students in Santiago, Chile (YPSA): study protocol for a randomized controlled trial , 2011, Trials.

[2]  Fiona Steele,et al.  Analysis of Multivariate Social Science Data, Second Edition , 2008 .

[3]  Dimitris Rizopoulos,et al.  ltm: An R Package for Latent Variable Modeling and Item Response Analysis , 2006 .

[4]  E. Walker,et al.  Diagnostic and Statistical Manual of Mental Disorders , 2013 .

[5]  W. W. Koczkodaj.  Statistically Accurate Evidence of Improved Error Rate by Pairwise Comparisons , 1996 .

[6]  K. Zou,et al.  Receiver-Operating Characteristic Analysis for Evaluating Diagnostic Tests and Predictive Models , 2007, Circulation.

[7]  Lutz Bornmann,et al.  The influence of the applicants’ gender on the modeling of a peer review process by using latent Markov models , 2009, Scientometrics.

[8]  Lucila Ohno-Machado,et al.  The use of receiver operating characteristic curves in biomedical informatics , 2005, J. Biomed. Informatics.

[9]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[10]  R. J. De Ayala,et al.  A Comparison of the Partial Credit and Graded Response Models in Computerized Adaptive Testing , 1992 .

[11]  E C Carterette,et al.  Visual and Auditory Information Processing in Children and Adults , 1967, Science.

[12]  Rolph E. Anderson,et al.  Multivariate Data Analysis (7th ed. , 2009 .

[13]  Janet B W Williams,et al.  Diagnostic and Statistical Manual of Mental Disorders , 2013 .

[14]  André Beauducel,et al.  On the Performance of Maximum Likelihood Versus Means and Variance Adjusted Weighted Least Squares Estimation in CFA , 2006 .

[15]  Wolfgang Glänzel,et al.  Grand challenges in data integration—state of the art and future perspectives: an introduction , 2016, Scientometrics.

[16]  W. W. Koczkodaj.  Testing the accuracy enhancement of pairwise comparisons by a Monte Carlo experiment , 1998 .

[17]  Ricardo Araya,et al.  Erratum to: Detecting depression among adolescents in Santiago, Chile: sex differences , 2013, BMC Psychiatry.

[18]  Ricardo Araya,et al.  Detecting depression among adolescents in Santiago, Chile: sex differences , 2013, BMC Psychiatry.

[19]  Gualberto Buela-Casal,et al.  What do the scientists think about the impact factor? , 2012, Scientometrics.

[20]  魏屹东,et al.  Scientometrics , 2018, Encyclopedia of Big Data.

[21]  Dimitrios Rizopoulos.  ltm: An R Package for Latent Variable Modeling and Item Response Theory Analyses , 2006 .

[22]  Willem E. Saris,et al.  Testing Structural Equation Models or Detection of Misspecifications? , 2009 .

[23]  Feng Li,et al.  A note on relevance of diagnostic classification and rating scales used in psychiatry , 2013, Comput. Methods Programs Biomed..

[24]  Philippe Le Moigne,et al.  Science as instrumentation. The case for psychiatric rating scales , 2012, Scientometrics.

[25]  Waldemar W. Koczkodaj,et al.  Improving the medical scale predictability by the pairwise comparisons method: Evidence from a clinical data study , 2012, Comput. Methods Programs Biomed..

[26]  Katarina Prpic,et al.  Changes of scientific knowledge production and research productivity in a transitional society , 2007, Scientometrics.

[27]  Konrad Kulakowski,et al.  On the quality evaluation of scientific entities in Poland supported by consistency-driven pairwise comparisons method , 2014, Scientometrics.

[28]  A. Beck,et al.  Beck Depression Inventory–II , 2011 .