Information mining and similarity computation for semi- / un-structured sentences from the social data

Abstract In recent years, with the development of the social Internet of Things (IoT), all kinds of data accumulated on the network. These data, which contain a lot of social information and opinions. However, these data are rarely fully analyzed, which is a major obstacle to the intelligent development of the social IoT. In this paper, we propose a sentence similarity analysis model to analyze the similarity in people’s opinions on the hot topics in social media and news pages. Most of these data are unstructured or semi-structured sentences, so the accuracy of sentence similarity analysis largely determines the model’s performance. For the purpose of improving accuracy, we propose a novel method of sentence similarity computation to extract the syntactic and semantic information of the semi-structured and unstructured sentences. We mainly consider the subjects, predicates and objects of sentence pairs, and use Stanford Parser to classify the dependency relation triples to calculate the syntactic and semantic similarity between two sentences. Finally, we verify the performance of the model with the Microsoft Research Paraphrase Corpus (MRPC), which consists of 4076 pairs of training sentences and 1725 pairs of test sentences, and most of the data came from the news of social data. Extensive simulations demonstrate that our method outperforms other state-of-the-art methods regarding the correlation coefficient and the mean deviation.