Developing e-Research Tools for the Analysis of Large-Scale Web Crawl Data

In this paper we describe the development of e-Research tools enabling remote access and analysis of large-scale web crawl data. The tools are being developed in the context of a planned research project titled the “.au Census”, the aim of which is to gain new insights into Australian commerce and society using data from large-scale crawls of the Australian public web.