MobileVisFixer: Tailoring Web Visualizations for Mobile Phones Leveraging an Explainable Reinforcement Learning Framework

We contribute MobileVisFixer, a new method to make visualizations more mobile-friendly. Although mobile devices have become the primary means of accessing information on the web, many existing visualizations are not optimized for small screens and can lead to a frustrating user experience. Currently, practitioners and researchers have to engage in a tedious and time-consuming process to ensure that their designs scale to screens of different sizes, and existing toolkits and libraries provide little support in diagnosing and repairing issues. To address this challenge, MobileVisFixer automates a mobile-friendly visualization re-design process with a novel reinforcement learning framework. To inform the design of MobileVisFixer, we first collected and analyzed SVG-based visualizations on the web, and identified five common mobile-friendly issues. MobileVisFixer addresses four of these issues on single-view Cartesian visualizations with linear or discrete scales by a Markov Decision Process model that is both generalizable across various visualizations and fully explainable. MobileVisFixer deconstructs charts into declarative formats, and uses a greedy heuristic based on Policy Gradient methods to find solutions to this difficult, multi-criteria optimization problem in reasonable time. In addition, MobileVisFixer can be easily extended with the incorporation of optimization algorithms for data visualizations. Quantitative evaluation on two real-world datasets demonstrates the effectiveness and generalizability of our method.

[1]  David S. Ebert,et al.  Visual Analytics on Mobile Devices for Emergency Response , 2007, 2007 IEEE Symposium on Visual Analytics Science and Technology.

[2]  Wilmot Li,et al.  Techniques for Flexible Responsive Visualization Design , 2020, CHI.

[3]  Jeffrey Heer,et al.  Reverse‐Engineering Visualizations: Recovering Visual Encodings from Chart Images , 2017, Comput. Graph. Forum.

[4]  Gorjan Alagic,et al.  #p , 2019, Quantum information & computation.

[5]  Michael Stonebraker,et al.  Beagle : Automated Extraction and Interpretation of Visualizations from the Web , 2017 .

[6]  Çagatay Demiralp,et al.  Data2Vis: Automatic Generation of Data Visualizations Using Sequence-to-Sequence Recurrent Neural Networks , 2018, IEEE Computer Graphics and Applications.

[7]  Tim Kraska,et al.  VizML: A Machine Learning Approach to Visualization Recommendation , 2018, CHI.

[8]  Bongshin Lee,et al.  Charticulator: Interactive Construction of Bespoke Chart Layouts , 2019, IEEE Transactions on Visualization and Computer Graphics.

[9]  Kanit Wongsuphasawat,et al.  Voyager 2: Augmenting Visual Analysis with Partial View Specifications , 2017, CHI.

[10]  Sean A. Munson,et al.  When (ish) is My Bus?: User-centered Visualizations of Uncertainty in Everyday, Mobile Predictive Systems , 2016, CHI.

[11]  Bongshin Lee,et al.  Visualizing Ranges over Time on Mobile Phones: A Task-Based Crowdsourced Evaluation , 2019, IEEE Transactions on Visualization and Computer Graphics.

[12]  Bongshin Lee,et al.  A Comparative Evaluation of Animation and Small Multiples for Trend Visualization on Mobile Phones , 2019, IEEE Transactions on Visualization and Computer Graphics.

[13]  Guoliang Li,et al.  Making data visualization more efficient and effective: a survey , 2019, The VLDB Journal.

[14]  Phil McMinn,et al.  Automated repair of layout cross browser issues using search-based techniques , 2017, ISSTA.

[15]  Bongshin Lee,et al.  ChartSense: Interactive Data Extraction from Chart Images , 2017, CHI.

[16]  Sergey Levine,et al.  DeepMimic , 2018, ACM Trans. Graph..

[17]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[18]  Hanspeter Pfister,et al.  Evaluating ‘Graphical Perception’ with CNNs , 2018, IEEE Transactions on Visualization and Computer Graphics.

[19]  Maneesh Agrawala,et al.  Converting Basic D3 Charts into Reusable Style Templates , 2016, IEEE Transactions on Visualization and Computer Graphics.

[20]  M. Sheelagh T. Carpendale,et al.  Data Changes Everything: Challenges and Opportunities in Data Visualization Design Handoff , 2019, IEEE Transactions on Visualization and Computer Graphics.

[21]  David S. Rosenberg,et al.  Scatteract: Automated Extraction of Data from Scatter Plots , 2017, ECML/PKDD.

[22]  Niklas Elmqvist,et al.  Visualizing for the Non‐Visual: Enabling the Visually Impaired to Use Visualization , 2019, Comput. Graph. Forum.

[23]  Aaron Hertzmann,et al.  Learning Layouts for Single-PageGraphic Designs , 2014, IEEE Transactions on Visualization and Computer Graphics.

[24]  Brian L. Price,et al.  DVQA: Understanding Data Visualizations via Question Answering , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Bongshin Lee,et al.  Mobile Data Visualization (Dagstuhl Seminar 19292) , 2019, Dagstuhl Reports.

[26]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[27]  Taehoon Kim,et al.  Quantifying Generalization in Reinforcement Learning , 2018, ICML.

[28]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[29]  Vijay V. Vazirani,et al.  Approximation Algorithms , 2001, Springer Berlin Heidelberg.

[30]  Bongshin Lee,et al.  Data Visualization on Mobile Devices , 2018, CHI Extended Abstracts.

[31]  Arvind Satyanarayan,et al.  Vega-Lite: A Grammar of Interactive Graphics , 2018, IEEE Transactions on Visualization and Computer Graphics.

[32]  Cheri Mullins,et al.  Responsive, mobile app, mobile first: untangling the UX design web in practical experience , 2015, SIGDOC.

[33]  Jonathan C. Roberts,et al.  Visualization beyond the Desktop--the Next Big Thing , 2014, IEEE Computer Graphics and Applications.

[34]  Jeffrey Heer,et al.  ReVision: automated classification, analysis and redesign of chart images , 2011, UIST.

[35]  Jeffrey Heer,et al.  Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco , 2018, IEEE Transactions on Visualization and Computer Graphics.

[36]  Pat Hanrahan,et al.  An Empirical Model of Slope Ratio Comparisons , 2012, IEEE Transactions on Visualization and Computer Graphics.

[37]  Ann Nowé,et al.  Multi-objective reinforcement learning using sets of pareto dominating policies , 2014, J. Mach. Learn. Res..

[38]  Maneesh Agrawala,et al.  Deconstructing and restyling D3 visualizations , 2014, UIST.

[39]  Martin A. Riedmiller,et al.  Evaluation of Policy Gradient Methods and Variants on the Cart-Pole Benchmark , 2007, 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning.

[40]  Phil McMinn,et al.  Automatic Visual Verification of Layout Failures in Responsively Designed Web Pages , 2019, 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST).

[41]  C. Lee Giles,et al.  Automatic Extraction of Data from Bar Charts , 2015, K-CAP.

[42]  Andrew W. Moore,et al.  Variable Resolution Discretization in Optimal Control , 2002, Machine Learning.

[43]  Phil McMinn,et al.  Automated Repair of Mobile Friendly Problems in Web Pages , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[44]  Maneesh Agrawala,et al.  Searching the Visual Style and Structure of D3 Visualizations , 2019, IEEE Transactions on Visualization and Computer Graphics.

[45]  Robert Babuska,et al.  A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[46]  Phil McMinn,et al.  Automated layout failure detection for responsive web pages without an explicit oracle , 2017, ISSTA.

[47]  Ronald A. DeVore,et al.  Some remarks on greedy algorithms , 1996, Adv. Comput. Math..

[48]  Yang Chen,et al.  Visualizing Large Time-series Data on Very Small Screens , 2017, EuroVis.

[49]  Frédo Durand,et al.  Learning Visual Importance for Graphic Designs and Data Visualizations , 2017, UIST.

[50]  Jeffrey Heer,et al.  Extracting and Retargeting Color Mappings from Bitmap Images of Visualizations , 2018, IEEE Transactions on Visualization and Computer Graphics.

[51]  Arvind Satyanarayan,et al.  Reactive Vega: A Streaming Dataflow Architecture for Declarative Interactive Visualization , 2016, IEEE Transactions on Visualization and Computer Graphics.

[52]  Luca Chittaro,et al.  Visualizing information on mobile devices , 2006, Computer.

[53]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[54]  Joseph A. Paradiso,et al.  The gesture recognition toolkit , 2014, J. Mach. Learn. Res..

[55]  Philip Bachman,et al.  Deep Reinforcement Learning that Matters , 2017, AAAI.

[56]  Theresa-Marie Rhyne,et al.  Reaching Broader Audiences With Data Visualization , 2020, IEEE Computer Graphics and Applications.

[57]  Pat Hanrahan,et al.  Show Me: Automatic Presentation for Visual Analysis , 2007, IEEE Transactions on Visualization and Computer Graphics.

[58]  Jeffrey Heer,et al.  Beyond Heuristics: Learning Visualization Design , 2018, ArXiv.

[59]  Phil McMinn,et al.  Automatic Detection of Potential Layout Faults Following Changes to Responsive Web Pages (N) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).