Automatic web accessibility metrics: Where we are and where we can go

The fact that several web accessibility metrics exist may be evidence of a lack of a comparison framework that highlights how well they work and for what purposes they are appropriate. In this paper we aim at formulating such a framework, demonstrating that it is feasible, and showing the findings we obtained when we applied it to seven existing automatic accessibility metrics. The framework encompasses validity, reliability, sensitivity, adequacy and complexity of metrics in the context of four scenarios where the metrics can be used. The experimental demonstration of the viability of the framework is based on applying seven published metrics to more than 1500 web pages and then operationalizing the notions of validity-as-conformance, adequacy and complexity. Our findings lead us to conclude that the Web Accessibility Quantitative Metric, Page Measure and Web Accessibility Barrier are the metrics that achieve the highest levels of quality (out of the seven that we examined). Finally, since we did not analyse reliability, sensitivity and validity-in-use, this paper provides guidance to address them in what are new research avenues.

[1]  Bambang Parmanto,et al.  Accessibility of Internet websites through time , 2004, Assets '04.

[2]  Carole A. Goble,et al.  The travails of visually impaired web travellers , 2000, HYPERTEXT '00.

[3]  Giorgio Brajnik,et al.  Validity and reliability of web accessibility guidelines , 2009, Assets '09.

[4]  Giorgio Brajnik,et al.  SAMBA: a semi-automatic method for measuring barriers of accessibility , 2007, Assets '07.

[5]  Markel Vigo,et al.  Enriching web information scent for blind users , 2009, Assets '09.

[6]  Yeliz Yesilada,et al.  Testability and validity of WCAG 2.0: the expertise effect , 2010, ASSETS '10.

[7]  Gilbert Cockton,et al.  Capability, acceptability and aspiration for: Collecting accessibility data with prototypes , 2005 .

[8]  Luisa Mich,et al.  Evaluating and designing Web site quality , 2003 .

[9]  Giorgio Brajnik,et al.  Beyond Conformance: The Role of Accessibility Evaluation Methods , 2008, WISE Workshops.

[10]  Giorgio Brajnik,et al.  Effects of sampling methods on web accessibility evaluations , 2007, Assets '07.

[11]  Jakob Nielsen,et al.  Homepage Usability: 50 Websites Deconstructed , 2001 .

[12]  Peter Brusilovsky,et al.  Adaptive Navigation Support , 2007, The Adaptive Web.

[13]  Mervyn Jack,et al.  Proceedings of HCI 2004 , 2004 .

[14]  Marco Roccetti,et al.  Metrics for accessibility on the Vamolà project , 2009, W4A.

[15]  John M. Slatin,et al.  Maximum Accessibility: Making Your Web Site More Usable for Everyone , 2002 .

[16]  Elizabeth Burd,et al.  Towards More Mature Web Maintenance Practices for Accessibility , 2007, 2007 9th IEEE International Workshop on Web Site Evolution.

[17]  Wolfgang L. Zagler,et al.  Computers Helping People with Special Needs, 12th International Conference, ICCHP 2010, Vienna, Austria, July 14-16, 2010, Proceedings, Part II , 2010, ICCHP.

[18]  Atiwong Suchato,et al.  Quantitative evaluation for web accessibility with respect to disabled groups , 2009, W4A.

[19]  Henry R. ole Kulet.  Is it possible , 1971 .

[20]  Gustavo Rossi,et al.  Measuring Web Application Quality with WebQEM , 2002, IEEE Multim..

[21]  Nils Ulltveit-Moe,et al.  Interpreting Results from Large Scale Automatic Evaluation of Web Accessibility , 2006, ICCHP.

[22]  Luís Carriço,et al.  The impact of accessibility assessment in macro scale universal usability studies of the web , 2008, W4A '08.

[23]  Gilbert Cockton,et al.  Falsification testing for usability inspection method assessment , 2004 .

[24]  Constantine Stephanidis,et al.  # 2001 Kluwer Academic Publishers. Printed in the Netherlands. Adaptive Techniques for Universal Access , 1999 .

[25]  Jennifer Mankoff,et al.  Is your web page accessible?: a comparative study of methods for assessing web page accessibility for the blind , 2005, CHI.

[26]  Rebecca Matson,et al.  Barriers to use: usability and content accessibility on the Web's most popular sites , 2000, CUU '00.

[27]  Gregg C. Vanderheiden,et al.  Web content accessibility guidelines 1.0 , 2001, INTR.

[28]  Takayuki Watanabe.  Experimental evaluation of usability and accessibility of heading elements , 2007, W4A '07.

[29]  Myriam Arrue,et al.  The use of guidelines to automatically verify Web accessibility , 2004, Universal Access in the Information Society.

[30]  Fernando Alonso,et al.  On the testability of WCAG 2.0 for beginners , 2010, W4A.

[31]  Giorgio Brajnik,et al.  Comparing accessibility evaluation tools: a method for tool effectiveness , 2004, Universal Access in the Information Society.

[32]  Luis Olsina,et al.  Specifying Quality Requirements for the Web 2.0 Applications , 2008, IWWOST@ICWE.

[33]  Gregg C. Vanderheiden,et al.  Web Content Accessibility Guidelines 1.0 W3C DRAFT Proposed Recommendation , 1999 .

[34]  William M. K. Trochim,et al.  Research methods knowledge base , 2001 .

[35]  Hironobu Takagi,et al.  Collaborative web accessibility improvement: challenges and possibilities , 2009, Assets '09.

[36]  Morten Goodwin Olsen,et al.  Is It Possible to Predict the Manual Web Accessibility Result Using the Automatic Result? , 2009, HCI.

[37]  Markel Vigo,et al.  Tool independence for the web accessibility quantitative metric , 2009, Disability and rehabilitation. Assistive technology.

[38]  Alfred Kobsa.  User Modeling and User-Adapted Interaction , 2005, User Modeling and User-Adapted Interaction.

[39]  Melody Y. Ivory,et al.  Search result exploration: a preliminary study of blind and sighted users' decision making and performance , 2004, CHI EA '04.

[40]  Elizabeth Burd,et al.  Tree-map visualisation for Web accessibility , 2005, 29th Annual International Computer Software and Applications Conference (COMPSAC'05).

[41]  Fernando Sánchez-Figueroa,et al.  Accessibility Metrics of Web Pages for Blind End-Users , 2003, ICWE.

[42]  Jozo J. Dujmovic.  A Method For Evaluation And Selection Of Complex Hardware And Software Systems , 1996, Int. CMG Conference.

[43]  Markel Vigo,et al.  Quantitative metrics for measuring web accessibility , 2007, W4A '07.

[44]  F. Thomas Eggemeier,et al.  Workload assessment methodology. , 1986 .

[45]  Junji Maeda,et al.  Accessibility designer: visualizing usability for the blind , 2004, Assets '04.

[46]  Hironobu Takagi,et al.  Proposing new metrics to evaluate web usability for the blind , 2005, CHI Extended Abstracts.

[47]  Bambang Parmanto,et al.  Metric for Web accessibility evaluation , 2005, J. Assoc. Inf. Sci. Technol..

[48]  Cynthia C. Shelly,et al.  Application of traditional software testing methodologies to web accessibility , 2010, W4A.

[49]  Gustavo Rossi,et al.  Web Engineering , 2001, Lecture Notes in Computer Science.

[50]  Carole A. Goble,et al.  Augmenting the mobility of profoundly blind Web travellers , 2005, New Rev. Hypermedia Multim..

[51]  Helen Petrie,et al.  The relationship between accessibility and usability of websites , 2007, CHI.

[52]  Takayuki Watanabe.  Experimental evaluation of usability and accessibility of heading elements , 2009, Disability and rehabilitation. Assistive technology.

[53]  Motoei Azuma.  SquaRE The next generation of the ISO/IEC 9126 and 14598 international standards series on software product quality , 2001 .

[54]  Alfred Kobsa,et al.  The Adaptive Web, Methods and Strategies of Web Personalization , 2007, The Adaptive Web.

[55]  Alfred Kobsa.  Adapting Web Information to Disabled and Elderly Users , 1999, WebNet.

[56]  Kasper Hornbæk,et al.  A Study of the Evaluator Effect in Usability Testing , 2008, Hum. Comput. Interact..

[57]  Klaus Opwis,et al.  Beyond web content accessibility guidelines: Design of enhanced text user interfaces for blind internet users , 2008, Int. J. Hum. Comput. Stud..

[58]  Alfred Kobsa,et al.  User-tailored web accessibility evaluations , 2007, HT '07.