Towards making videos accessible for low vision screen magnifier users

People with low vision who use screen magnifiers to interact with computing devices find it very challenging to interact with dynamically changing digital content such as videos, since they do not have the luxury of time to manually move, i.e., pan the magnifier lens to different regions of interest (ROIs) or zoom into these ROIs before the content changes across frames. In this paper, we present SViM, a first of its kind screen-magnifier interface for such users that leverages advances in computer vision, particularly video saliency models, to identify salient ROIs in videos. SViM's interface allows users to zoom in/out of any point of interest, switch between ROIs via mouse clicks and provides assistive panning with the added flexibility that lets the user explore other regions of the video besides the ROIs identified by SViM. Subjective and objective evaluation of a user study with 13 low vision screen magnifier users revealed that overall the participants had a better user experience with SViM over extant screen magnifiers, indicative of the former's promise and potential for making videos accessible to low vision screen magnifier users.

[1]  Haibin Ling,et al.  Revisiting Video Saliency Prediction in the Deep Learning Era , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Hanqiu Sun,et al.  Video Saliency Prediction Using Spatiotemporal Residual Attentive Networks , 2020, IEEE Transactions on Image Processing.

[3]  Kristen Grauman,et al.  Making 360° Video Watchable in 2D: Learning Videography for Click Free Viewing , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[5]  Pei-Yu Chi,et al.  MixT: automatic generation of step-by-step mixed media tutorials , 2012, CHI Extended Abstracts.

[6]  Pierre Kornprobst,et al.  Navisio: Towards an integrated reading aid system for low vision patients , 2008 .

[7]  I. V. Ramakrishnan,et al.  SteeringWheel: A Locality-Preserving Magnification Interface for Low Vision Web Browsing , 2018, CHI.

[8]  Jon Froehlich,et al.  Design of an Augmented Reality Magnification Aid for Low Vision Users , 2018, ASSETS.

[9]  Tie Liu,et al.  DeepVS: A Deep Learning Based Video Saliency Prediction Approach , 2018, ECCV.

[10]  Chokri Ben Amar,et al.  Transfer learning with deep networks for saliency prediction in natural video , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[11]  Kim-Phuong L. Vu,et al.  How Screen Magnification with and Without Word-Wrapping Affects the User Experience of Adults with Low Vision , 2017, AHFE.

[12]  Kristen Grauman,et al.  Pano2Vid: Automatic Cinematography for Watching 360° Videos , 2017, WICED@Eurographics.

[13]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[14]  Petros Maragos,et al.  SUSiNet: See, Understand and Summarize It , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[15]  Aykut Erdem,et al.  Spatio-Temporal Saliency Networks for Dynamic Saliency Prediction , 2016, IEEE Transactions on Multimedia.

[16]  Makoto J. Hirayama.  A book reading magnifier for low vision persons on smartphones and tablets , 2018, 2018 International Workshop on Advanced Image Technology (IWAIT).

[17]  Qi Zhao,et al.  SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[18]  Jeffrey P. Bigham.  Making the web easier to see with opportunistic accessibility improvement , 2014, UIST.

[19]  Ali Borji,et al.  Saliency Prediction in the Deep Learning Era: An Empirical Investigation , 2018, ArXiv.

[20]  Noel E. O'Connor,et al.  Shallow and Deep Convolutional Networks for Saliency Prediction , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  James J. Clark,et al.  Going from Image to Video Saliency: Augmenting Image Salience with Dynamic Attentional Push , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Gang Luo,et al.  Magnifying Smartphone Screen Using Google Glass for Low-Vision Users , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[23]  Poorna Kushalnagar,et al.  Multi-view platform: an accessible live classroom viewing approach for low vision students , 2011, ASSETS '11.

[24]  Asha Iyer,et al.  Components of bottom-up gaze allocation in natural images , 2005, Vision Research.

[25]  J. V. Bradley.  Complete Counterbalancing of Immediate Sequential Effects in a Latin Square Design , 1958 .

[26]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[27]  Yuhang Zhao,et al.  CueSee: exploring visual cues for people with low vision to facilitate a visual search task , 2016, UbiComp.

[28]  Gordon Wetzstein,et al.  Saliency in VR: How Do People Explore Virtual Environments? , 2016, IEEE Transactions on Visualization and Computer Graphics.

[29]  James Norris,et al.  CamBlend: an object focused collaboration tool , 2012, CHI.

[30]  Ming-Hsuan Yang,et al.  Semantic-Driven Generation of Hyperlapse from 360 Degree Video , 2018, IEEE Transactions on Visualization and Computer Graphics.

[31]  American Foundation for the Blind , 1967 .

[32]  Hugo Larochelle,et al.  Recurrent Mixture Density Network for Spatiotemporal Visual Attention , 2016, ICLR.

[33]  Yuhang Zhao,et al.  ForeSee: A Customizable Head-Mounted Vision Enhancement System for People with Low Vision , 2015, ASSETS.

[34]  Nicole C Ross,et al.  Preliminary Evaluation of Two Digital Image Processing Strategies for Head-Mounted Magnification for Low Vision Patients , 2019, Translational Vision Science & Technology.

[35]  Michael Christen,et al.  The Effect of Magnification and Contrast on Reading Performance in Different Types of Simulated Low Vision , 2016, Journal of eye movement research.

[36]  Meredith Ringel Morris,et al.  SeeingVR: A Set of Tools to Make Virtual Reality More Accessible to People with Low Vision , 2019, CHI.

[37]  Jon Froehlich,et al.  Augmented Reality Magnification for Low Vision Users with the Microsoft Hololens and a Finger-Worn Camera , 2017, ASSETS.