A Vision on Mining Visual Logs of Software

In this paper, we define visual logs of a software system as data capturing the interactions between its users and its graphic user interfaces (GUI) such as screenshots and screen recordings. We expect that mining such visual logs could be useful for many software engineering tasks: bug reproducing and debugging, automated GUI testing, user interface design, question answering of common software usages, etc. Toward that vision, we propose a core framework for mining visual logs of software. This framework focuses on detecting GUI elements and changes in visual logs, removing users' private data, recognizing user interactions with GUI elements, and learning GUI usage patterns. We also presented a small study on characteristics of GUI elements in mobile apps. The findings from this study suggest several heuristics to design techniques for recognizing GUI elements and interactions.