Big data for scientific research and discovery

With data volumes expanding beyond the Petabyte and Exabyte levels across many scientific disciplines, the role of big data for scientific research is becoming increasingly apparent: the massive data processing has become valuable for scientific research. The term big data is not only a buzzword and a marketing tool, but also it can provide invaluable help for scientific research and discovery through the data-intensive scientific discovery paradigm. ‘Big Data for Development: Challenges & Opportunities’ by United Nations Global Pulse, an initiative of the Secretary-General on big data, suggesting that projects/ programs of big data research promote a national strategy, and pointing out the essential role of big data for the development of society as a whole, including science and technology, economics, and decision-making. It is noteworthy that UN Global Pulse launched the Big Data Climate Challenge as part of the Secretary-General’s 2014 Climate Summit, which provides data-driven evidence of the impacts of climate change. The UN spokesperson Stephane Dujurric said, ‘This initiative will help build public understanding of how big data can reveal critical insights for strengthening resilience and mitigating emissions.’ Numerous examples of big data’s contribution to scientific discoveries have been identified, especially for big interdisciplinary research, such as Digital Earth and Global Change. Unprecedentedly, large datasets generated, sensed, and harvested from experiments, observations, and simulations have brought great opportunities for making scientific progress for two reasons: (1) Huge datasets will serve as important inputs and will support adjustment and validation of current theories for large scientific problems, thus leading to new findings. A good example is the new paradigm of ‘big data meets big models’ for large inverse problems. (2) Massive datasets themselves are able to provide endless sources of new knowledge without modeling the scientific phenomena. This has been characterized as the ‘Fourth Paradigm’ – data-intensive scientific discovery. There is no doubt that big data will significantly change the way scientific discoveries are made. Scientists must be prepared to welcome a new age in which digital data will play an important role and might dominate the methodologies for scientific research. Scientists will inevitably face a number of challenges before the aforementioned opportunities are realized. Considering the technology level, there are mainly three challenges to overcome before big data can significantly contribute to scientific discoveries. The first challenge is for data processing infrastructures and platforms. New generations of software and hardware with high performance, high scalability, and high efficiency are required for sensing, storing, and computing big data. The second challenge faced by scientists mining big data are the processing algorithms. It is highly encouraged to develop new models and algorithms for efficiently locating interesting findings in the context of vast, unstructured, heterogeneous, nonlinear, nonsteady, highdimensional datasets. The third challenge is the methodologies for linking the processing to the discovery: how can big data be integrated with the processes of research International Journal of Digital Earth, 2015 Vol. 8, No. 1, 1–2, http://dx.doi.org/10.1080/17538947.2015.1015942