A decision support system for usability evaluation of web-based information systems

In this study, a decision support system (DSS) for usability assessment and design of web-based information systems (WIS) is proposed. It employs three machine learning methods (support vector machines, neural networks, and decision trees) and a statistical technique (multiple linear regression) to reveal the underlying relationships between the overall WIS usability and its determinative factors. A sensitivity analysis on the predictive models is performed and a new metric, criticality index, is devised to identify the importance ranking of the determinative factors. Checklist items with the highest and the lowest contribution to the usability performance of the WIS are specified by means of the criticality index. The most important usability problems for the WIS are determined with the help of a pseudo-Pareto analysis. A case study through a student information system at Fatih University is carried out to validate the proposed DSS. The proposed DSS can be used to decide which usability problems to focus on so as to improve the usability and quality of WIS.

[1]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[2]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[3]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[4]  R. Teas,et al.  Expectations, Performance Evaluation, and Consumers’ Perceptions of Quality , 1993 .

[5]  F. Buttle.  SERVQUAL: review, critique, research agenda , 1996 .

[6]  Emin Babakus,et al.  An empirical assessment of the SERVQUAL scale , 1992 .

[7]  Heng-Li Yang,et al.  A three-stage model of requirements elicitation for Web-based information systems , 2003, Ind. Manag. Data Syst..

[8]  Steven C. Wheelwright,et al.  Forecasting methods and applications. , 1979 .

[9]  A. Parasuraman,et al.  SERVQUAL: A multiple-item scale for measuring consumer perceptions of service quality. , 1988 .

[10]  Jeff Sauro,et al.  A method to standardize usability metrics into a single score , 2005, CHI.

[11]  James R. Lewis,et al.  Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies , 2002, Int. J. Hum. Comput. Interact..

[12]  G. V. Kass.  An Exploratory Technique for Investigating Large Quantities of Categorical Data , 1980 .

[13]  Stefano Tarantola,et al.  Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models , 2004 .

[14]  B Efron,et al.  Statistical Data Analysis in the Computer Age , 1991, Science.

[15]  Min Xie,et al.  Measuring web-based service quality , 2002 .

[16]  Richard T. Vidgen,et al.  Measuring Web site quality improvements: a case study of the forum on strategic management knowledge exchange , 2003, Ind. Manag. Data Syst..

[17]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[18]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[19]  Nigel Bevan,et al.  Quality in use: Meeting user needs for quality , 1999, J. Syst. Softw..

[20]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[21]  Benjamin Keevil,et al.  Measuring the usability index of your Web site , 1998, SIGDOC '98.

[22]  G. W. Davis,et al.  Sensitivity analysis in neural net solutions , 1989, IEEE Trans. Syst. Man Cybern..

[23]  V. Zeithaml,et al.  E-S-QUAL A Multiple-Item Scale for Assessing Electronic Service Quality , 2004 .

[24]  Nigel Bevan,et al.  Measuring usability as quality of use , 1995, Software Quality Journal.

[25]  S. Stoeva,et al.  WebUse: an approach for web usability evaluation , 2003 .

[26]  Leon A. Kappelman,et al.  Standard user interface in e-commerce sites , 2003, Ind. Manag. Data Syst..

[27]  Mary Corbett,et al.  SUMI: the Software Usability Measurement Inventory , 1993, Br. J. Educ. Technol..

[28]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[29]  Asil Oztekin,et al.  UseLearn: A novel checklist and usability evaluation method for eLearning systems by criticality metric analysis , 2010 .

[30]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[31]  K. Pearson,et al.  ON THE GENERALISED PROBABLE ERROR IN MULTIPLE NORMAL CORRELATION , 1908 .

[32]  Jacob Cohen,et al.  Applied multiple regression/correlation analysis for the behavioral sciences , 1979 .

[33]  A. Saltelli,et al.  Making best use of model evaluations to compute sensitivity indices , 2002 .

[34]  Steven A. Taylor,et al.  Measuring Service Quality: A Reexamination and Extension , 1992 .

[35]  Jeff Sauro,et al.  Correlations among prototypical usability metrics: evidence for the construct of usability , 2009, CHI.

[36]  Alison Dean,et al.  Analysing service quality in the hospitality industry , 1999 .

[37]  Efraim Turban,et al.  Decision Support and Business Intelligence Systems (8th Edition) , 2006 .

[38]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[39]  R. Kirk.  Experimental Design: Procedures for the Behavioral Sciences , 1970 .

[40]  Steven A. Taylor,et al.  Servperf versus Servqual: Reconciling Performance-Based and Perceptions-Minus-Expectations Measurement of Service Quality , 1994 .

[41]  Selim Zaim,et al.  UWIS: An assessment methodology for usability of web-based information systems , 2009, J. Syst. Softw..

[42]  Gavriel Salvendy,et al.  A proposed index of usability: A method for comparing the relative usability of different software systems , 1997, Behav. Inf. Technol..

[43]  A. Parasuraman,et al.  Delivering quality service : balancing customer perceptions and expectations , 1990 .

[44]  Kasper Hornbæk,et al.  Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? , 2000, CHI.

[45]  T. Hassard,et al.  Applied Linear Regression , 2005 .

[46]  David L. Olson,et al.  Advanced Data Mining Techniques , 2008 .

[47]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[48]  Jose C. Principe,et al.  Neural and adaptive systems , 2000 .

[49]  J. Carman.  Consumer perceptions of service quality: an assessment of the SERVQUAL dimensions , 1990 .