Efficient quality-driven source selection from massive data sources

This paper first studies data source selection problem from massive data sources based on data quality.The algorithm takes data quality and the relation to the query into consideration to achieve efficiency and effectiveness data source selection.Experimental results demonstrate that our method can scale to millions of data sources and preforms pretty efficiently. The query based on massive database is time-consuming and difficult. And the uneven quality of data source makes the multiple source selection more challenging. The low-quality data source can even make the result of the information unexpected. How to efficiently select quality-driven data sources on massive database remains a hard problem. In this paper, we study the efficient source selection problem on massive data set considering the quality of data sources. Our approach evaluates the quality of data source and balances the limitation of resources and the completeness of data source. For data source selection for a specific query, our method could select the data sources with the number of keywords larger than a given threshold. And the selected sources are ranked according to the values of information in data sources. Experimental results demonstrate that our method can scale to millions of data sources and perform pretty efficiently.

[1]  Philip S. Yu,et al.  Truth Discovery with Multiple Conflicting Information Providers on the Web , 2008, IEEE Trans. Knowl. Data Eng..

[2]  Dan Roth,et al.  Knowing What to Believe (when you already know something) , 2010, COLING.

[3]  Ashwin Machanavajjhala,et al.  Information integration over time in unreliable and uncertain environments , 2012, WWW.

[4]  Felix Naumann,et al.  Data fusion , 2009, CSUR.

[5]  Divesh Srivastava,et al.  Characterizing and selecting fresh data sources , 2014, SIGMOD Conference.

[6]  Sergey Brin,et al.  Reprint of: The anatomy of a large-scale hypertextual web search engine , 2012, Comput. Networks.

[7]  Allan Borodin,et al.  Link analysis ranking: algorithms, theory, and experiments , 2005, TOIT.

[8]  Divesh Srivastava,et al.  Less is More: Selecting Sources Wisely for Integration , 2012, Proc. VLDB Endow..

[9]  Ling Liu,et al.  TrustMe: anonymous management of trust relationships in decentralized P2P systems , 2003, Proceedings Third International Conference on Peer-to-Peer Computing (P2P2003).

[10]  Dan Roth,et al.  Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence Making Better Informed Trust Decisions with Generalized Fact-Finding , 2022 .

[11]  Jon Kleinberg,et al.  Authoritative sources in a hyperlinked environment , 1999, SODA '98.

[12]  Serge Abiteboul,et al.  Corroborating information from disagreeing views , 2010, WSDM '10.

[13]  Felix Naumann,et al.  Data Fusion – Resolving Data Conflicts for Integration , 2009 .

[14]  Alun D. Preece,et al.  Incorporating Domain-Specific Information Quality Constraints into Database Queries , 2009, JDIQ.

[15]  Jiaheng Lu,et al.  Efficient Merging and Filtering Algorithms for Approximate String Searches , 2008, 2008 IEEE 24th International Conference on Data Engineering.

[16]  Xiaoxin Yin,et al.  Semi-supervised truth discovery , 2011, WWW.

[17]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[18]  Anthony K. H. Tung,et al.  Effective keyword-based selection of relational databases , 2007, SIGMOD '07.

[19]  Divesh Srivastava,et al.  Truth Discovery and Copying Detection in a Dynamic World , 2009, Proc. VLDB Endow..

[20]  Maria-Esther Vidal,et al.  Using Quality of Data Metadata for Source Selection and Ranking , 2000, WebDB.

[21]  Philip S. Yu,et al.  Truth Discovery with Multiple Conflicting Information Providers on the Web , 2007, IEEE Transactions on Knowledge and Data Engineering.

[22]  Hector Garcia-Molina,et al.  The Eigentrust algorithm for reputation management in P2P networks , 2003, WWW '03.

[23]  Shazia Wasim Sadiq,et al.  Data Quality Aware Queries in Collaborative Information Systems , 2009, APWeb/WAIM.

[24]  Ee-Peng Lim,et al.  Quality-aware collaborative question answering: methods and evaluation , 2009, WSDM '09.