In the course of web research it is often necessary to estimate the creation datetime for web resources (in the general case, this value can only be estimated). While it is feasible to manually establish likely datetime values for small numbers of resources, this becomes infeasible if the collection is large. We present "carbon date", a simple web application that estimates the creation date for a URI by polling a number of sources of evidence and returning a machine-readable structure with their respective values. To establish a likely datetime, we poll bitly for the first time someone shortened the URI, topsy for the first time someone tweeted the URI, a Memento aggregator for the first time it appeared in a public web archive, Google's time of last crawl, and the Last-Modified HTTP response header of the resource itself. We also examine the backlinks of the URI as reported by Google and apply the same techniques for the resources that link to the URI. We evaluated our tool on a gold standard data set of 1200 URIs in which the creation date was manually verified. We were able to estimate a creation date for 75.90% of the resources, with 32.78% having the correct value. Given the different nature of the URIs, the union of the various methods produces the best results. While the Google last crawl date and topsy account for nearly 66% of the closest answers, eliminating the web archives or Last-Modified from the results produces the largest overall negative impact on the results. The carbon date application is available for download or use via a web API.
[1]
Michael L. Nelson,et al.
Agreeing to disagree: search engines and their public interfaces
,
2007,
JCDL '07.
[2]
Adam Jatowt,et al.
Detecting age of page content
,
2007,
WIDM '07.
[3]
Masahiro Inoue,et al.
Noise robust detection of the emergence and spread of topics on the web
,
2012,
TempWeb '12.
[4]
Lars R. Clausen,et al.
Concerning Etags and Datestamps
,
2004
.
[5]
Michael L. Nelson,et al.
How much of the web is archived?
,
2011,
JCDL '11.
[6]
Michael L. Nelson,et al.
Search engines and their public interfaces: which apis are the most synchronized?
,
2007,
WWW '07.
[7]
Michael L. Nelson,et al.
Losing My Revolution: How Many Resources Shared on Social Media Have Been Lost?
,
2012,
TPDL.
[8]
Sotiris Ioannidis,et al.
we.b: the web of short urls
,
2011,
WWW.
[9]
Herbert Van de Sompel,et al.
Memento: Time Travel for the Web
,
2009,
ArXiv.