Proposing a new local density estimation outlier detection algorithm: an empirical case study on flow pattern experiments

Outlier or anomaly detection is an important branch of data analysis that becomes a crucial task in many application domains. Data objects which significantly dissimilar and inconsistent from the rest of the data objects are referred to as an outlier. In this paper, a new approach, called LDBAD (Local Density-Based Abnormal Detector), is proposed to discover useful irregular patterns hidden in the collected data sets. This method aims to find local abnormal data objects, which are characterized through three proposed measurements: local distance, local density, and Influenced outlierness degree. The performance of the proposed approach is evaluated on flow pattern experiments along a 180 degrees sharp bend channel with and without a T-shaped spur dike. Flow velocity components are collected using 3D velocimeter Vectrino. The analysis shows that the novel outlier detection method is effective and applicable to find outlier objects. Moreover, some feed-forward neural network velocity prediction models are created to demonstrate the necessity and advantages of outlier detection in flow pattern experiments. The results show that the accuracy of created models has been increased by removing outliers from the measurements.

[1]  Velocity correction of the Janus configuration laser Doppler velocimeter , 2013 .

[2]  Rommel N. Carvalho,et al.  Applying clustering and AHP methods for evaluating suspect healthcare claims , 2017, J. Comput. Sci..

[3]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD 2000.

[4]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[5]  Andreas Theissler,et al.  Detecting known and unknown faults in automotive systems using ensemble-based anomaly detection , 2017, Knowl. Based Syst..

[6]  Felix Naumann,et al.  Data fusion , 2009, CSUR.

[7]  J. Tolvi,et al.  Genetic algorithms for outlier detection and variable selection in linear regression models , 2004, Soft Comput..

[8]  Xue-feng Yan.  Multivariate outlier detection based on self-organizing map and adaptive nonlinear map and its application , 2011 .

[9]  Nicola Zannone,et al.  An anomaly analysis framework for database systems , 2015, Comput. Secur..

[10]  Raheb Bagherpour,et al.  Forecasting ground vibration due to rock blasting: a hybrid intelligent approach using support vector regression and fuzzy C-means clustering , 2018, Engineering with Computers.

[11]  Hong Shi,et al.  Three-way k-means: integrating k-means and three-way decision , 2019, International Journal of Machine Learning and Cybernetics.

[12]  Charu C. Aggarwal,et al.  Outlier Analysis , 2013, Springer New York.

[13]  H. Ghassemi,et al.  Temporal and spatial characteristics of wave energy in the Persian Gulf based on the ERA5 reanalysis dataset , 2019, Energy.

[14]  Oral Alan,et al.  Thresholds based outlier detection approach for mining class outliers: An empirical case study on software measurement datasets , 2011, Expert Syst. Appl..

[15]  Su Yang,et al.  LDBOD: A novel local distribution based outlier detector , 2008, Pattern Recognit. Lett..

[16]  Maciej Łuczak,et al.  Hierarchical clustering of time series data with parametric derivative dynamic time warping , 2016 .

[17]  Shuchita Upadhyaya,et al.  Outlier Detection: Applications And Techniques , 2012 .

[18]  J. Westerweel,et al.  Universal outlier detection for PIV data , 2005 .

[20]  Mahdi Hasanipanah,et al.  A new combination of artificial neural network and K-nearest neighbors models to predict blast-induced ground vibration and air-overpressure , 2016, Engineering with Computers.

[21]  Xiaowei Yang,et al.  Robust least squares support vector machine based on recursive outlier elimination , 2010, Soft Comput..

[22]  Mohammad Vaghefi,et al.  Application of artificial neural networks to predict flow velocity in a 180° sharp bend with and without a spur dike , 2020, Soft Comput..

[23]  Mohammad Vaghefi,et al.  Detection of Outlier in 3D Flow Velocity Collection in an Open-Channel Bend Using Various Data Mining Techniques , 2018, Iranian Journal of Science and Technology, Transactions of Civil Engineering.

[24]  M. Vaghefi,et al.  A Comparison among Data Mining Algorithms for Outlier Detection using Flow Pattern Experiments , 2017 .

[25]  Ji Zhang,et al.  Detecting anomalies from high-dimensional wireless network data streams: a case study , 2011, Soft Comput..

[26]  Mohiuddin Ahmed,et al.  A survey of network anomaly detection techniques , 2016, J. Netw. Comput. Appl..

[27]  Mehdi Nikoo,et al.  Determination of compressive strength of concrete using Self Organization Feature Map (SOFM) , 2013, Engineering with Computers.

[28]  Flip Korn,et al.  Influence sets based on reverse nearest neighbor queries , 2000, SIGMOD 2000.

[29]  P. Santhi Thilagam,et al.  Mining social networks for anomalies: Methods and challenges , 2016, J. Netw. Comput. Appl..

[30]  H. Ghassemi,et al.  Data mining models to predict ocean wave energy flux in the absence of wave records , 2017 .

[31]  H. Ghassemi,et al.  Wind energy potential assessment in the Persian Gulf: A spatial and temporal analysis , 2020 .

[32]  Shadi Aljawarneh,et al.  Anomaly-based intrusion detection system through feature selection analysis and building hybrid efficient model , 2017, J. Comput. Sci..

[33]  Hassan Ghassemi,et al.  Outlier Detection in Ocean Wave Measurements by Using Unsupervised Data Mining Methods , 2018 .

[34]  Hassan Ghassemi,et al.  Prediction of the hydrodynamic performance and cavitation volume of the marine propeller using gene expression programming , 2019, Ships and Offshore Structures.

[35]  Nicola Greggio,et al.  Anomaly Detection in IDSs by means of unsupervised greedy learning of finite mixture models , 2018, Soft Comput..

[36]  Anthony K. H. Tung,et al.  Ranking Outliers Using Symmetric Neighborhood Relationship , 2006, PAKDD.

[37]  Maurizio Filippone,et al.  A comparative evaluation of outlier detection algorithms: Experiments and analyses , 2018, Pattern Recognit..

[38]  Morteza Gharib,et al.  Universal outlier detection for particle image velocimetry (PIV) and particle tracking velocimetry (PTV) data , 2010 .

[39]  Vladimir Nikora,et al.  Despiking Acoustic Doppler Velocimeter Data , 2002 .

[40]  Fan Ming-hui.  Review of Outlier Detection , 2006 .

[41]  Yumin Chen,et al.  Neighborhood outlier detection , 2010, Expert Syst. Appl..

[42]  Frank Klawonn,et al.  A Novel Approach to Noise Clustering for Outlier Detection , 2006, Soft Comput..

[43]  Pyoung Won Kim,et al.  Adaptive switching filter for impulse noise removal in digital content , 2018, Soft Comput..