Dealing with data in large end-use load metering projects: Data quality and data access

Hourly end-use electrical consumption data collected for the Bonneville Power Administration and the United States Department of Energy as part of the End-Use Load Conservation Assessment Program (ELCAP) are subjected to extensive data processing prior to entry in the data archive. Due to the extremely large size of the data set, all data processing activities must be highly automated. Two of the most important data processing activities are data quality checks and pre-aggregation of the hourly data to daily and monthly levels. The data quality checks allow both the rapid identification of problems and the encoding of a concise indicator of data quality for individual data records. A detailed procedure based on conservation of energy principles has been developed for data quality checks on data collected as part of ELCAP. The pre-aggregation processing makes the most commonly used aggregations (daily, monthly, monthly profile) directly available for analysis, freeing the analyst to concentrate on the information content of the data. One of the most important lessons learned in ELCAP is that up to 50% of the time and money necessary to produce an analytical product may be spent in just preparing the data for analysis. Automation of the pre-aggregation process significantlymore » reduces the time needed for data preparation. This paper describes the need for automated data quality checks and pre-aggregation processing and discusses the specific checks that are performed on all ELCAP data as it is collected, processed, and archived for eventual analysis. 8 refs., 3 figs.« less