Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications

Usability testing of software applications developed for mobile devices is an emerging research area that faces a variety of challenges due to unique features of mobile devices, limited bandwidth, unreliability of wireless networks, as well as changing context (environmental factors). Traditional guidelines and methods used in usability testing of desktop applications may not be directly applicable to a mobile environment. Therefore, it is essential to develop and adopt appropriate research methodologies that can evaluate the usability of mobile applications. The contribution of this article is to propose a generic framework for conducting usability tests for mobile applications through discussing research questions, methodologies, and usability attributes. The article provides an overview of existing mobil application usability studies and discusses major research questions that have been investigated. Then, it proposes a generic framework and provides detailed guidelines on how to conduct such usability studies.

[1]  George Buchanan,et al.  Improving Web Interaction on Small Displays , 1999, Comput. Networks.

[2]  Rachelle S. Heller,et al.  Using a theoretical multimedia taxonomy framework , 2001, JERC.

[3]  Timo Laakko,et al.  Two approaches to bringing Internet services to WAP devices , 2000, Comput. Networks.

[4]  John R. Smith,et al.  Scalable multimedia delivery for pervasive computing , 1999, MULTIMEDIA '99.

[5]  Leysia Palen,et al.  Beyond the handset: designing for wireless communications usability , 2002, TCHI.

[6]  Andreas Paepcke,et al.  Seeing the whole in parts: text summarization for web browsing on handheld devices , 2001, WWW '01.

[7]  A M Wichansky,et al.  Usability testing in 2000 and beyond , 2000, Ergonomics.

[8]  Mary Czerwinski,et al.  A fisheye calendar interface for PDAs: providing overviews for small displays , 2003, CHI Extended Abstracts.

[9]  Franca Garzotto,et al.  Model-based heuristic evaluation of hypermedia usability , 1998, AVI '98.

[10]  Kristian Kiili,et al.  Evaluating WAP usability: "what usability?" , 2002, Proceedings. IEEE International Workshop on Wireless and Mobile Technologies in Education.

[11]  Michael J. Smith,et al.  Usability evaluation and interface design : cognitive engineering, intelligent agents and virtual reality , 2001 .

[12]  Marianne G. Petersen.  Towards usability evaluation of multimedia applications , 1998, CROS.

[13]  Andreas Paepcke,et al.  Efficient web browsing on handheld devices using page and form summarization , 2002, TOIS.

[14]  I. Scott MacKenzie,et al.  Theoretical upper and lower bounds on typing speed using a stylus and a soft keyboard , 1995, Behav. Inf. Technol..

[15]  Anne Suwita,et al.  Evaluating the usability of the Siemens C10 Mobile Phone going beyond common practice in industry , 2005, Personal Technologies.

[16]  Elliot Soloway,et al.  Pocket PiCoMap: a case study in designing and assessing a handheld concept mapping tool for learners , 2003, CHI '03.

[17]  Gregory D. Abowd,et al.  A Conceptual Framework and a Toolkit for Supporting the Rapid Prototyping of Context-Aware Applications , 2001, Hum. Comput. Interact..

[18]  Leysia Palen,et al.  Voice-mail diary studies for naturalistic data capture under mobile conditions , 2002, CSCW '02.

[19]  Joseph Krajcik,et al.  Scaffolding in the small: designing educational supports for concept mapping on handheld computers , 2002, CHI Extended Abstracts.

[20]  Teija Vainio,et al.  Developing 3D information systems for mobile users: some usability issues , 2002, NordiCHI '02.

[21]  Martina Ziefle,et al.  The influence of user expertise and phone complexity on performance, ease of use and learnability of different mobile phones , 2002, Behav. Inf. Technol..

[22]  Lars Erik Holmquist,et al.  POWERVIEW: Using Information Links and Information Views to Navigate and Visualize Information on Small Displays , 2000, HUC.

[23]  Maria Ebling,et al.  On the contributions of different empirical data in usability testing , 2000, DIS '00.

[24]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[25]  Shumin Zhai,et al.  Top-down learning strategies: can they facilitate stylus keyboard learning? , 2004, Int. J. Hum. Comput. Stud..

[26]  Stephen R. Gulliver,et al.  Pervasive and standalone computing: the perceptual effects of variable multimedia quality , 2004, Int. J. Hum. Comput. Stud..

[27]  Masood Masoodian,et al.  An Empirical Study of Textual and Graphical Travel Itinerary Visualization using Mobile Phones , 2003, AUIC.

[28]  Michael J. Albers,et al.  Web design issues when searching for information in a small screen display , 2001, SIGDOC '01.

[29]  Dongsong Zhang,et al.  Delivery of Personalized and Adaptive Content to Mobile Devices: A Framework and Enabling Technology , 2003, Commun. Assoc. Inf. Syst..

[30]  Roy D. Pea,et al.  Functioning in the wireless classroom , 2004, The 2nd IEEE International Workshop on Wireless and Mobile Technologies in Education, 2004. Proceedings..

[31]  Andrew Sears,et al.  Understanding the Relation Between Network Quality of Service and the Usability of Distributed Multimedia Documents , 2000, Hum. Comput. Interact..

[32]  Bill N. Schilit,et al.  Digestor: Device-Independent Access to the World Wide Web , 1997, Comput. Networks.

[33]  Shawn Xia Zhang.  A high-performance soft keyboard for mobile systems , 1998 .

[34]  Lars Erik Holmquist,et al.  WEST: a Web browser for small terminals , 1999, UIST '99.

[35]  Franca Garzotto,et al.  HDM—a model-based approach to hypertext application design , 1993, TOIS.

[36]  Ismo Rakkolainen,et al.  A 3D City Info for mobile users , 2001, Comput. Graph..

[37]  Upkar Varshney,et al.  Mobile Commerce: Framework, Applications and Networking Support , 2002, Mob. Networks Appl..

[38]  Mikael Goldstein,et al.  Towards an improved readability on mobile devices: evaluating adaptive rapid serial visual presentation , 2002, Interact. Comput..

[39]  Luca Chittaro,et al.  Evaluating Interface Design Choices on WAP Phones: Navigation and Selection , 2002, Personal and Ubiquitous Computing.

[40]  Kori Inkpen Quinn,et al.  GeneyTM: designing a collaborative activity for the palmTM handheld computer , 2001, CHI.

[41]  Avi Parush,et al.  Web navigation structures in cellular phones: the depth/breadth trade-off issue , 2004, Int. J. Hum. Comput. Stud..

[42]  Kasper Hornbæk,et al.  Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? , 2000, CHI.

[43]  Elliot Soloway,et al.  An engineering process for constructing scaffolded work environments to support student inquiry: a case study in history , 2001, Proceedings IEEE International Conference on Advanced Learning Technologies.

[44]  I. Scott MacKenzie,et al.  The design and evaluation of a high-performance soft keyboard , 1999, CHI '99.

[45]  Franca Garzotto,et al.  Adding multimedia collections to the Dexter Model , 1994, ECHT '94.

[46]  Jan Stage,et al.  Experimental Evaluation of Techniques for Usability Testing of Mobile Systems in a Laboratory Setting , 2003 .

[47]  Pavel Slavík,et al.  PDA based navigation system for a 3D environment , 2001, Comput. Graph..

[48]  Jinwoo Kim,et al.  An empirical study of the use contexts and usability problems in mobile Internet , 2002, Proceedings of the 35th Annual Hawaii International Conference on System Sciences.

[49]  Jakob Nielsen,et al.  Chapter 4 – The Usability Engineering Lifecycle , 1993 .

[50]  Franca Garzotto,et al.  Hypermedia design, analysis, and evaluation issues , 1995, CACM.

[51]  Carolyn R. Watters,et al.  A comparison of simple hierarchy and grid metaphors for option layouts on small-size screens , 2004, Int. J. Hum. Comput. Stud..

[52]  Scott P. Robertson,et al.  Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 1991 .

[53]  I. Scott MacKenzie,et al.  Text entry using soft keyboards , 1999, Behav. Inf. Technol..

[54]  Jan Stage,et al.  New techniques for usability evaluation of mobile systems , 2004, Int. J. Hum. Comput. Stud..

[55]  Mike Sharples,et al.  The Design and Implementation of a Mobile Learning Resource , 2002, Personal and Ubiquitous Computing.

[56]  George Buchanan,et al.  Improving mobile internet usability , 2001, WWW '01.