Comprehensive Automated Quality Assurance of Daily Surface Observations

This paper describes a comprehensive set of fully automated quality assurance (QA) procedures for observations of daily surface temperature, precipitation, snowfall, and snow depth. The QA procedures are being applied operationally to the Global Historical Climatology Network (GHCN)-Daily dataset. Since these data are used for analyzing and monitoring variations in extremes, the QA system is designed to detect as many errors as possible while maintaining a low probability of falsely identifying true meteorological events as erroneous. The system consists of 19 carefully evaluated tests that detect duplicate data, climatological outliers, and various inconsistencies (internal, temporal, and spatial). Manual review of random samples of the values flagged as errors is used to set the threshold for each procedure such that its falsepositive rate, or fraction of valid values identified as errors, is minimized. In addition, the tests are arranged in a deliberate sequence in which the performance of the later checks is enhanced by the error detection capabilities of the earlier tests. Based on an assessment of each individual check and a final evaluation for each element, the system identifies 3.6 million (0.24%) of the more than 1.5 billion maximum/minimum temperature, precipitation, snowfall, and snow depth values in GHCN-Daily as errors, has a false-positive rate of 1%22%, and is effective at detecting both the grossest errors as well as more subtle inconsistencies among elements.

[1]  K. Trenberth,et al.  Observations: Surface and Atmospheric Climate Change , 2007 .

[2]  J. Lawrimore,et al.  Extreme Weather Records , 2007 .

[3]  D. Wilks,et al.  Automated quality control procedure for the {open_quotes}water equivalent of snow on the ground{close_quotes} measurement , 1995 .

[4]  Paul Roebber,et al.  Improving Snowfall Forecasting by Diagnosing Snow Density , 2003 .

[5]  T. Owen,et al.  A Deterministic Approach to the Validation of Historical Daily Temperature and Precipitation Data from the Cooperative Network. , 1992 .

[6]  Thomas C. Peterson,et al.  Global historical climatology network (GHCN) quality control of monthly temperature data , 1998 .

[7]  Qi Hu,et al.  Quality control of daily meteorological data in China, 1951–2000: a new dataset , 2004 .

[8]  M. Clark,et al.  Characteristics of Snowfall over the Eastern Half of the United States and Relationships with Principal Modes of Low-Frequency Atmospheric Variability , 1998 .

[9]  George H. Taylor,et al.  Observer Bias in Daily Precipitation Measurements at United States Cooperative Network Stations , 2007 .

[10]  K. Kunkel,et al.  An Expanded Digital Daily Database for Climatic Resources Applications in the Midwestern United States. , 1998 .

[11]  D. Lister,et al.  The development of a new dataset of Spanish Daily Adjusted Temperature Series (SDATS) (1850–2003) , 2006 .

[12]  D. Legates,et al.  Evaluating the use of “goodness‐of‐fit” Measures in hydrologic and hydroclimatic model validation , 1999 .

[13]  David R. Legates,et al.  The Accuracy of United States Precipitation Data , 1994 .

[14]  Some Concerns when Using Data from the Cooperative Weather Station Networks:A Nebraska Case Study , 2005 .

[15]  Jinsheng You,et al.  An Improved QC Process for Temperature in the Daily Cooperative Weather Observations , 2007 .

[16]  J. Scott Greene,et al.  The Comprehensive Pacific Rainfall Database , 2008 .

[17]  Barry E. Goodison,et al.  Accuracy of Canadian Snow Gage Measurements , 1978 .

[18]  Steve Goddard,et al.  Performance of Quality Assurance Procedures for an Applied Climate Information System , 2005 .

[19]  B. Brasnett,et al.  A Global Analysis of Snow Depth for Numerical Weather Prediction , 1999 .

[20]  John R. Lanzante,et al.  Resistant, Robust and Non-Parametric Techniques for the Analysis of Climate Data: Theory and Examples, Including Applications to Historical Radiosonde Station Data , 1996 .

[21]  David A. Robinson,et al.  EVALUATION OF THE COLLECTION, ARCHIVING AND PUBLICATION OF DAILY SNOW DATA IN THE UNITED STATES , 1989 .

[22]  D. Easterling.  United States Historical Climatology Network Daily Temperature and Precipitation Data (1871-1997) , 2002 .

[23]  J. Lawrimore,et al.  Compilation, Adjudication, and Publication , 2007 .

[24]  David Robinson.  The United States cooperative climate-observing systems: reflections and recommendations , 1990 .

[25]  Russell S. Vose,et al.  Robust Automated Quality Assurance of Radiosonde Temperatures , 2008 .

[26]  Michael A. Palecki,et al.  Trend Identification in Twentieth-Century U.S. Snowfall: The Challenges , 2007 .

[27]  David R. Easterling,et al.  Quality Control of Pre-1948 Cooperative Observer Network Data , 2005 .

[28]  Jinsheng You,et al.  Sensitivity Analysis of Quality Assurance Using the Spatial Regression Approach—A Case Study of the Maximum/Minimum Air Temperature , 2005 .

[29]  M. Janis.  Observation-Time-Dependent Biases and Departures for Daily Minimum and Maximum Air Temperatures. , 2002 .

[30]  N. Nicholls.  Long-term climate monitoring and extreme events , 1995 .

[31]  Henry F. Diaz,et al.  The Quality Control of Long-Term Climatological Data Using Objective Data Analysis , 1995 .

[32]  J. V. Revadekar,et al.  Global observed changes in daily climate extremes of temperature and precipitation , 2006 .

[33]  Robert G. Quayle,et al.  A Review of Cooperative Temperature Data Validation , 1990 .

[34]  R. Vose,et al.  Large-scale changes in observed daily maximum and minimum temperatures: Creation and analysis of a new gridded data set , 2006 .

[35]  Jinsheng You,et al.  Quality Control of Weather Data during Extreme Events , 2006 .

[36]  C. W. Richardson,et al.  Evaluating the Adequacy of Simulating Maximum and Minimum Daily Air Temperature with the Normal Distribution , 2002 .

[37]  Klaus Wolter,et al.  Trimming Problems and Remedies in COADS , 1997 .

[38]  J. R. Wallis,et al.  A daily hydroclimatological data set for the continental United States , 1991 .

[39]  Henry F. Diaz,et al.  Creating a Serially Complete, National Daily Time Series of Temperature and Precipitation for the Western United States , 2000 .

[40]  Matthew J. Menne,et al.  Strategies for evaluating quality assurance procedures , 2008 .