Who and what links to the Internet Archive

The Internet Archive’s (IA) Wayback Machine is the largest and oldest public Web archive and has become a significant repository of our recent history and cultural heritage. Despite its importance, there has been little research about how it is discovered and used. Based on Web access logs, we analyze what users are looking for, why they come to IA, where they come from, and how pages link to IA. We find that users request English pages the most, followed by the European languages. Most human users come to Web archives because they do not find the requested pages on the live Web. About 65 % of the requested archived pages no longer exist on the live Web. We find that more than 82 % of human sessions connect to the Wayback Machine via referrals from other Web sites, while only 15 % of robots have referrers. Most of the links (86 %) from Websites are to individual archived pages at specific points in time, and of those 83 % no longer exist on the live Web. Finally, we find that users who come from search engines browse more pages than users who come from external Web sites.

[1]  Morimichi Nishigaki,et al.  ArchivesZ : Visualizing Archival Collections , 2007 .

[2]  KoehlerWallace.  Web page change and persistence---a four-year longitudinal study , 2002 .

[3]  Charles Teddlie,et al.  Mixed Methods Sampling A Typology With Examples , 2016 .

[4]  Brewster Kahle,et al.  Preserving the Internet , 1997 .

[5]  Michele C. Weigle,et al.  Visualizing digital collections at archive-it , 2012, JCDL '12.

[6]  Wallace Koehler,et al.  Web page change and persistence - A four-year longitudinal study , 2002, J. Assoc. Inf. Sci. Technol..

[7]  Michael L. Nelson,et al.  Just-in-time recovery of missing web pages , 2006, HYPERTEXT '06.

[8]  Giovanna Castellano,et al.  LODAP: a log data preprocessor for mining web browsing patterns , 2007 .

[9]  Maria Dolores C. Tongco,et al.  Purposive Sampling as a Tool for Informant Selection , 2007 .

[10]  Herbert Van de Sompel,et al.  Analyzing the Persistence of Referenced Web Resources with Memento , 2011, ArXiv.

[11]  P. Sumathi,et al.  Novel Pre-Processing Technique for Web Log Mining by Removing Global Noise, Cookies and Web Robots , 2012 .

[12]  Haibin Liu,et al.  Combined mining of Web server logs and web contents for classifying user navigation patterns and predicting users' future requests , 2007, Data Knowl. Eng..

[13]  Herbert Van de Sompel,et al.  Profiling web archive coverage for top-level domain and content language , 2013, International Journal on Digital Libraries.

[14]  Ravi Kumar,et al.  A characterization of online browsing behavior , 2010, WWW '10.

[15]  Marios D. Dikaiakos,et al.  Web robot detection: A probabilistic reasoning approach , 2009, Comput. Networks.

[16]  Mitchell Whitelaw.  Exploring Archival Collections with Interactive Visualisation , 2009 .

[17]  Swapna S. Gokhale,et al.  Web robot detection techniques: overview and limitations , 2010, Data Mining and Knowledge Discovery.

[18]  Hiroshi Esaki,et al.  The impact of residential broadband traffic on Japanese ISP backbones , 2005, CCRV.

[19]  Michael L. Nelson,et al.  Access patterns for robots and humans in web archives , 2013, JCDL '13.

[20]  Miguel Costa,et al.  Characterizing Search Behavior in Web Archives , 2011, TWAW.

[21]  Steven J. M. Jones,et al.  Circos: an information aesthetic for comparative genomics. , 2009, Genome research.

[22]  Jaideep Srivastava,et al.  Web usage mining: discovery and applications of usage patterns from Web data , 2000, SKDD.

[23]  Mike Thelwall,et al.  A fair history of the Web? Examining country balance in the Internet Archive , 2004 .

[24]  Brad Tofel.  ‘Wayback’ for Accessing Web Archives , 2007 .

[25]  Jaideep Srivastava,et al.  Data Preparation for Mining World Wide Web Browsing Patterns , 1999, Knowledge and Information Systems.

[26]  Zdravko Markov,et al.  Comprar Data Mining the Web: Uncovering Patterns in Web Content, Structure, and Usage | Daniel T. Larose | 9780471666554 | Wiley , 2007 .

[27]  Yannis Manolopoulos,et al.  CLEAR: a credible method to evaluate website archivability , 2013, iPRES.

[28]  Michael L. Nelson,et al.  How much of the web is archived? , 2011, JCDL '11.

[29]  Ralf D. Brown,et al.  Selecting and Weighting N-Grams to Identify 1100 Languages , 2013, TSD.

[30]  Elad Yom-Tov,et al.  Enhancing digital libraries using missing content analysis , 2008, JCDL '08.

[31]  Herbert Van de Sompel,et al.  HTTP Framework for Time-Based Access to Resource States - Memento , 2013, RFC.

[32]  Gregg G. Van Ryzin,et al.  Cluster Analysis as a Basis for Purposive Sampling of Projects in Case Study Evaluations , 1995 .

[33]  Vipin Kumar,et al.  Discovery of Web Robot Sessions Based on their Navigational Patterns , 2004, Data Mining and Knowledge Discovery.

[34]  Nivio Ziviani,et al.  Finding what is missing from a digital library: A case study in the Computer Science field , 2009, Inf. Process. Manag..

[35]  Peter G. Anick.  Using terminological feedback for web search refinement: a log-based study , 2003, SIGIR.

[36]  Andrei Z. Broder,et al.  Sic transit gloria telae: towards an understanding of the web's decay , 2004, WWW '04.

[37]  C. Lee Giles,et al.  What's there and what's not?: focused crawling for missing documents in digital libraries , 2005, Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '05).

[38]  Zdravko Markov,et al.  Data mining the web - uncovering patterns in web content, structure, and usage , 2007 .

[39]  Marios D. Dikaiakos,et al.  An investigation of web crawler behavior: characterization and metrics , 2005, Comput. Commun..

[40]  Theint Theint Aye,et al.  Web log cleaning for mining of web usage patterns , 2011, 2011 3rd International Conference on Computer Research and Development.

[41]  Herbert Van de Sompel,et al.  Profiling web archive coverage for top-level domain and content language , 2013, International Journal on Digital Libraries.

[42]  Myra Spiliopoulou,et al.  A Framework for the Evaluation of Session Reconstruction Heuristics in Web-Usage Analysis , 2003, INFORMS J. Comput..

[43]  Brigitte Trousse,et al.  Advanced data preprocessing for intersites Web usage mining , 2004, IEEE Intelligent Systems.

[44]  K. Sudheer Reddy,et al.  Preprocessing the web server logs: an illustrative approach for effective usage mining , 2012, SOEN.

[45]  Michael L. Nelson,et al.  Carbon dating the web: estimating the age of web resources , 2013, WWW '13 Companion.

[46]  James E. Pitkow,et al.  Characterizing Browsing Strategies in the World-Wide Web , 1995, Comput. Networks ISDN Syst..

[47]  Michael L. Nelson,et al.  Object Persistence and Availability in Digital Libraries , 2002, D Lib Mag..

[48]  Mário J. Silva,et al.  Understanding the Information Needs of Web Archive Users , 2010 .

[49]  Herbert Van de Sompel,et al.  Memento: Time Travel for the Web , 2009, ArXiv.