Performance Model's development: A Novel Approach encompassing Ontology-Based Data Access and Visual Analytics

The quantitative evaluation of research is currently carried out by means of indicators calculated on data extracted and integrated by analysts who elaborate them by creating illustrative tables and plots of results. In this approach, the robustness of the metrics used and the possibility for users of the metrics to intervene in the evaluation process are completely neglected. We propose a new approach which is able to move forward, from indicators’ development to an interactive performance model’s development. It combines the advantages of the ontology-based data access paradigm with the flexibility and robustness of a visual analytics environment putting the consumer/stakeholder at the centre of the evaluation. A detailed description of such an approach is presented in the paper. The approach is illustrated and evaluated trough a comprehensive user’s study that proves the added capabilities and the benefits that a user of performance models can have by using this approach.

[1]  Evgeny Kharlamov,et al.  Ontology Based Data Access in Statoil , 2017, J. Web Semant..

[2]  Maurizio Lenzerini,et al.  Ontology-Based Data Access: The Experience at the Italian Department of Treasury , 2013, CAiSE Industrial Track.

[3]  Carsten Binnig,et al.  Evaluating Visual Data Analysis Systems: A Discussion Report , 2018, HILDA@SIGMOD.

[4]  Cinzia Daraio,et al.  A Framework for the Assessment of Research and Its Impacts , 2017, J. Data Inf. Sci..

[5]  Léopold Simar,et al.  Advanced Robust and Nonparametric Methods in Efficiency Analysis: Methodology and Applications , 2007 .

[6]  Fernando Roda,et al.  UNiCS - The Ontology for Research and Innovation Policy Making , 2018, FOIS.

[7]  Alfred Inselberg,et al.  Parallel Coordinates: Visual Multidimensional Geometry and Its Applications , 2003, KDIR.

[8]  Domenico Lembo,et al.  Easy OWL Drawing with the Graphol Visual Ontology Language , 2016, KR.

[9]  W. Cooper,et al.  Data Envelopment Analysis: A Comprehensive Text with Models, Applications, References and DEA-Solver Software , 1999 .

[10]  Lorna Elizabeth Wildgaard,et al.  An Overview of Author-Level Indicators of Research Performance , 2019, Springer Handbook of Science and Technology Indicators.

[11]  Diego Calvanese,et al.  Ontop: Answering SPARQL queries over relational databases , 2016, Semantic Web.

[12]  Abraham Charnes,et al.  Measuring the efficiency of decision making units , 1978 .

[13]  Tiziana Catarci,et al.  An Ontology Based Visual Tool for Query Formulation Support , 2004, OTM Workshops.

[14]  Maurizio Lenzerini,et al.  Data integration for research and innovation policy: an Ontology-Based Data Management approach , 2015, Scientometrics.

[15]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[16]  Georg von Mayr.  Die Gesetzmäßigkeit im Gesellschaftsleben : statistische Studien , 1877 .

[17]  A. Charnes,et al.  Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis , 1984 .

[18]  Cinzia Daraio,et al.  Econometric Approaches to the Measurement of Research Productivity , 2019, Springer Handbook of Science and Technology Indicators.

[19]  Giuseppe Santucci,et al.  CLAIRE: A combinatorial visual analytics system for information retrieval evaluation , 2018, Inf. Process. Manag..

[20]  Monica Scannapieco,et al.  Using Ontologies for Official Statistics: The Istat Experience , 2017, ICWE Workshops.

[21]  Diego Calvanese,et al.  Description Logic Framework for Information Integration , 1998, KR.

[22]  Luciano Floridi,et al.  The Fourth Revolution: How the infosphere is reshaping human reality , 2014 .

[23]  Giuseppe Santucci,et al.  Visualization and analysis of schema and instances of ontologies for improving user tasks and knowledge discovery , 2019, J. Comput. Lang..

[24]  Marco Angelini,et al.  Vulnus: Visual Vulnerability Analysis for Network Security , 2019, IEEE Transactions on Visualization and Computer Graphics.

[25]  Ian Horrocks,et al.  OptiqueVQS: A visual query system over ontologies for industry , 2018, Semantic Web.

[26]  András Schubert,et al.  All Along the h-Index-related Literature: A Guided Tour , 2019, Springer Handbook of Science and Technology Indicators.

[27]  Maurizio Lenzerini,et al.  The advantages of an Ontology-Based Data Management approach: openness, interoperability and data quality , 2016, Scientometrics.

[28]  Giuseppe Santucci,et al.  Cyber situational awareness: from geographical alerts to high-level management , 2017, J. Vis..

[29]  Wolfgang Glänzel,et al.  Grand challenges in data integration—state of the art and future perspectives: an introduction , 2016, Scientometrics.

[30]  Maurizio Lenzerini,et al.  Developing Ontology-based Data Management for the Italian Public Debt , 2014, SEBD.

[31]  Gunnar Sivertsen,et al.  Developing Current Research Information Systems (CRIS) as Data Sources for Studies of Research , 2019, Springer Handbook of Science and Technology Indicators.

[32]  Maurizio Lenzerini,et al.  Sapientia: the Ontology of Multi-dimensional Research Assessment , 2015, ISSI.

[33]  Enrique Herrera-Viedma,et al.  Science Mapping Analysis Software Tools: A Review , 2019, Springer Handbook of Science and Technology Indicators.

[34]  Cinzia Daraio,et al.  Beyond university rankings? Generating new indicators on universities by linking data in open platforms , 2017, J. Assoc. Inf. Sci. Technol..

[35]  Maurizio Lenzerini,et al.  Challenges, Approaches and Solutions in Data Integration for Research and Innovation , 2019, Springer Handbook of Science and Technology Indicators.

[36]  Valerie Belton,et al.  Demystifying DEA — A Visual Interactive Approach Based on Multiple Criteria Analysis , 1993 .

[37]  Peter Haase,et al.  Optique: Zooming in on Big Data , 2015, Computer.

[38]  Diego Calvanese,et al.  The MASTRO system for ontology-based data access , 2011, Semantic Web.

[39]  Daniel A. Keim,et al.  Visual Analytics: Definition, Process, and Challenges , 2008, Information Visualization.