On the Feasibility of Unstructured Peer-to-Peer Information Retrieval

We consider the feasibility of web-scale search in an unstructured peer-to-peer network. Since the network is unstructured, any such search is probabilistic in nature. We therefore adopt a probably approximately correct (PAC) search framework. The accuracy of such a search is defined by the overlap between the set of documents retrieved by a PAC search and the set of documents retrieved by an exhaustive (deterministic) search of the network. For an accuracy of 90%, we theoretically determine the number of nodes each query must be sent to for three distributions of documents in the network, namely uniform, proportional and square root. We assume that the query distribution follows a power law and investigate how performance is affected by the scale factor. For various configurations, we estimate the global and local network traffic induced by the search. For a network of 1 million nodes, a query rate of 1000 queries per second, and assuming each node is capable of indexing 0.1% of the collection, our analysis indicates that the network traffic is less that 0.07% of global internet traffic.

[1]  Ingemar J. Cox,et al.  Probably Approximately Correct Search , 2009, ICTIR.

[2]  Robert Tappan Morris,et al.  Security Considerations for Peer-to-Peer Distributed Hash Tables , 2002, IPTPS.

[3]  Ion Stoica,et al.  Peer-to-Peer Systems II , 2003, Lecture Notes in Computer Science.

[4]  Ian H. Witten,et al.  Managing gigabytes (2nd ed.): compressing and indexing documents and images , 1999 .

[5]  Suresh Jagannathan,et al.  Search with probabilistic guarantees in unstructured peer-to-peer networks , 2005, Fifth IEEE International Conference on Peer-to-Peer Computing (P2P'05).

[6]  Farnam Jahanian,et al.  Internet inter-domain traffic , 2010, SIGCOMM '10.

[7]  Scott Shenker,et al.  Making gnutella-like P2P systems scalable , 2003, SIGCOMM '03.

[8]  Amy L. Murphy,et al.  An Evaluation and Comparison of Current Peer-to-Peer Full-Text Keyword Search Techniques , 2005, WebDB.

[9]  Jussi Kangasharju,et al.  Bubblestorm: resilient, probabilistic, and exhaustive peer-to-peer search , 2007, SIGCOMM '07.

[10]  Milad Shokouhi,et al.  Advances in Information Retrieval Theory, Second International Conference on the Theory of Information Retrieval, ICTIR 2009, Cambridge, UK, September 10-12, 2009, Proceedings , 2009, ICTIR.

[11]  Michael K. Reiter,et al.  Probabilistic quorum systems , 1997, PODC '97.

[12]  Ian H. Witten,et al.  Managing Gigabytes: Compressing and Indexing Documents and Images , 1999 .

[13]  Brian F. Cooper.  An Optimal Overlay Topology for Routing Peer-to-Peer Searches , 2005, Middleware.

[14]  Edith Cohen,et al.  Replication strategies in unstructured peer-to-peer networks , 2002, SIGCOMM.

[15]  Martina Zitterbart,et al.  Proceedings of the ACM SIGCOMM 2003 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, August 25-29, 2003, Karlsruhe, Germany , 2003, SIGCOMM.

[16]  Mario A. Nascimento,et al.  Improving Web search efficiency via a locality based static pruning method , 2005, WWW '05.

[17]  Ingemar J. Cox,et al.  Improving Query Correctness Using Centralized Probably Approximately Correct (PAC) Search , 2010, ECIR.

[18]  Peter Ingwersen,et al.  Developing a Test Collection for the Evaluation of Integrated Search , 2010, ECIR.

[19]  Peter Druschel,et al.  Peer-to-peer systems , 2010, Commun. ACM.

[20]  David R. Karger,et al.  On the Feasibility of Peer-to-Peer Web Indexing and Search , 2003, IPTPS.

[21]  Vern Paxson,et al.  Proceedings of the 2002 conference on Applications, technologies, architectures, and protocols for computer communications , 2002, SIGCOMM 2002.

[22]  Yong Yang,et al.  Performance of Full Text Search in Structured and Unstructured Peer-to-Peer Systems , 2006, Proceedings IEEE INFOCOM 2006. 25TH IEEE International Conference on Computer Communications.

[23]  Aristides Gionis,et al.  The impact of caching on search engines , 2007, SIGIR.