Data set quality in Machine Learning: Consistency measure based on Group Decision Making

Abstract Performance of Machine Learning models heavily depends on the quality of the training dataset. Among others, the quality of training data relies on the consistency of the labels assigned to similar items. Indeed, the labels should be coherently assigned (or collected) by avoiding inconsistencies for increasing the performance of the machine learning model. This study focuses on evaluating training data consistency for machine learning algorithms dealing with ranking problems, i.e., the Learning to Rank methods (LTR). This work defines a training data consistency measure based on the consensus value introduced in Group Decision Making. It investigates the statistical relationship between the proposed consistency measure and the performance of a deep neural network implementing an LTR method. This measure could drive data filtering at the training stage and guide model update decisions. Experimentation reveals a strong correlation between the proposed consistency measure and the performance of the model.

[1]  Ammar Belatreche,et al.  Evaluating machine learning classification for financial trading: An empirical approach , 2016, Expert Syst. Appl..

[2]  Sanjay Krishnan,et al.  ActiveClean: An Interactive Data Cleaning Framework For Modern Machine Learning , 2016, SIGMOD Conference.

[3]  Mario Piattini,et al.  A Data Quality in Use model for Big Data , 2016, Future Gener. Comput. Syst..

[4]  Yangyong Zhu,et al.  The Challenges of Data Quality and Data Quality Assessment in the Big Data Era , 2015, Data Sci. J..

[5]  G. Kesteven,et al.  The Coefficient of Variation , 1946, Nature.

[6]  Felix Bießmann,et al.  On Challenges in Machine Learning Model Management , 2018, IEEE Data Eng. Bull..

[7]  Divesh Srivastava,et al.  Data quality: The other face of Big Data , 2014, 2014 IEEE 30th International Conference on Data Engineering.

[8]  Sebastian Bruch,et al.  TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank , 2018, KDD.

[9]  Mimmo Parente,et al.  Time-aware adaptive tweets ranking through deep learning , 2017, Future Gener. Comput. Syst..

[10]  Wenbin Cai,et al.  Batch Mode Active Learning for Regression With Expected Model Change , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[11]  Tao Chen,et al.  All Versus One: An Empirical Comparison on Retrained and Incremental Machine Learning for Modeling Performance of Adaptable Software , 2019, 2019 IEEE/ACM 14th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS).

[12]  Tilmann Rabl,et al.  Continuous Deployment of Machine Learning Pipelines , 2019, EDBT.

[13]  Huan Liu,et al.  Consistency-based search in feature selection , 2003, Artif. Intell..

[14]  Gregory N. Hullender,et al.  Learning to rank using gradient descent , 2005, ICML.

[15]  Enrico Zio,et al.  A Novel Concept Drift Detection Method for Incremental Learning in Nonstationary Environments , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[16]  Azadeh Shakery,et al.  Query-dependent learning to rank for cross-lingual information retrieval , 2018, Knowledge and Information Systems.

[17]  Franco Scarselli,et al.  SortNet: Learning to Rank by a Neural Preference Function , 2011, IEEE Transactions on Neural Networks.

[18]  Abhinav Gupta,et al.  Learning from Noisy Large-Scale Datasets with Minimal Supervision , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Tie-Yan Liu,et al.  Learning to rank for information retrieval , 2009, SIGIR.

[20]  Danilo Ardagna,et al.  Context-aware data quality assessment for big data , 2018, Future Gener. Comput. Syst..

[21]  Koby Crammer,et al.  Pranking with Ranking , 2001, NIPS.

[22]  Tie-Yan Liu,et al.  Adapting ranking SVM to document retrieval , 2006, SIGIR.

[23]  Enrique Herrera-Viedma,et al.  A framework for context-aware heterogeneous group decision making in business processes , 2016, Knowl. Based Syst..

[24]  Luis Martínez-López,et al.  Group Decision Making: From Consistency to Consensus , 2007, MDAI.

[25]  Charles E. Brown.  Coefficient of Variation , 1998 .

[26]  M. Kendall.  A NEW MEASURE OF RANK CORRELATION , 1938 .

[27]  Vincenzo Loia,et al.  Drift-Aware Methodology for Anomaly Detection in Smart Grid , 2019, IEEE Access.

[28]  James M. Keller,et al.  Extending deep learning to new classes without retraining , 2020, Defense + Commercial Sensing.

[29]  Tie-Yan Liu,et al.  Learning to rank: from pairwise approach to listwise approach , 2007, ICML '07.

[30]  Alessandro Moschitti,et al.  Learning to Rank Short Text Pairs with Convolutional Deep Neural Networks , 2015, SIGIR.

[31]  Enrique Herrera-Viedma,et al.  A novel multi-criteria group decision-making method for heterogeneous and dynamic contexts using multi-granular fuzzy linguistic modelling and consensus measures , 2020, Inf. Fusion.

[32]  Dario Landa Silva,et al.  An evolutionary strategy with machine learning for learning to rank in information retrieval , 2018, Soft Comput..

[33]  Meikang Qiu,et al.  Retraining Strategy-Based Domain Adaption Network for Intelligent Fault Diagnosis , 2020, IEEE Transactions on Industrial Informatics.

[34]  Stan Sclaroff,et al.  Deep Metric Learning to Rank , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Hwanjo Yu,et al.  Improving top-K recommendation with truster and trustee relationship in user trust network , 2016, Inf. Sci..

[36]  Francisco Herrera,et al.  Some issues on consistency of fuzzy preference relations , 2004, Eur. J. Oper. Res..

[37]  Venkat N. Gudivada,et al.  Data Quality Considerations for Big Data and Machine Learning: Going Beyond Data Cleaning and Transformations , 2017 .

[38]  Enrique Herrera-Viedma,et al.  A decision support system to develop a quality management in academic digital libraries , 2015, Inf. Sci..

[39]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[40]  Dan Pei,et al.  Personalized re-ranking for recommendation , 2019, RecSys.

[41]  Francisco Herrera,et al.  A consensus model for multiperson decision making with different preference structures , 2002, IEEE Trans. Syst. Man Cybern. Part A.

[42]  Taghi Ebadi,et al.  Industrial Wastes Risk Ranking with TOPSIS, Multi Criteria Decision Making Method , 2017 .

[43]  Hong Liu,et al.  Cleaning Framework for BigData: An Interactive Approach for Data Cleaning , 2016, 2016 IEEE Second International Conference on Big Data Computing Service and Applications (BigDataService).

[44]  Zhi-Hua Zhou,et al.  A brief introduction to weakly supervised learning , 2018 .

[45]  Chia-Yu Lin,et al.  Hybrid Real-Time Matrix Factorization for Implicit Feedback Recommendation Systems , 2018, IEEE Access.

[46]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[47]  Michael Laskey,et al.  Statistical data cleaning for deep learning of automation tasks from demonstrations , 2017, 2017 13th IEEE Conference on Automation Science and Engineering (CASE).

[48]  Gerhard Nahler,et al.  Pearson Correlation Coefficient , 2020, Definitions.

[49]  Tie-Yan Liu,et al.  Directly optimizing evaluation measures in learning to rank , 2008, SIGIR.

[50]  Enrique Herrera-Viedma,et al.  A Self-Management Mechanism for Noncooperative Behaviors in Large-Scale Group Consensus Reaching Processes , 2018, IEEE Transactions on Fuzzy Systems.

[51]  Enrique Herrera-Viedma,et al.  Evaluating the information quality of Web sites: A methodology based on fuzzy computing with words , 2006, J. Assoc. Inf. Sci. Technol..

[52]  Yoram Singer,et al.  An Efficient Boosting Algorithm for Combining Preferences by , 2013 .

[53]  Praveen Chandar,et al.  An Information Retrieval Framework for Contextual Suggestion Based on Heterogeneous Information Network Embeddings , 2018, SIGIR.

[54]  Ehsan Sadeh,et al.  Selecting an Appropriate Express Railway Pavement System Using VIKOR Multi-Criteria Decision Making Model , 2018 .