Flexible Techniques for Storage and Analysis of Large Continuing Surveys

A common characteristic of continuing surveys is that they tend to have a long history of evolutionary development. Questions asked may change over time, and the accumulated body of data becomes increasingly large and complex. This characteristic leads to the need for computer systems capable of dealing with complexity while providing efficient processing and data storage. Two generalized computer systems are examined to determine their effectiveness in meeting the processing needs of large-scale continuing surveys. Both systems were developed with a major goal being the processing of large and complex collections of micro data. One of the systems, called RAPID, was developed by Statistics Canada for data storage and maintenance. The other, called TPL for Table Producing language, was developed in the United States by the Bureau of Labor Statistics to do data retrieval, reduction, and table generation. Recent work has resulted in the linking of these two systems. A file design is discussed which takes advantage of the special features of these systems and demonstrates that their combined use is weil-suited to the processing requirements of large-scale continuing surveys.