Adjustment for Bias Caused by Non-Response in Mailed Surveys

INCOMPLETENESS of returns in a mail survey usually implies a certain degree of bias in the results because a respondent's willingness to return a schedule is generally related to the nature of the item to be estimated from the survey. The bias may be either positive or negative, depending upon whether prospective respondents with large or with small quantities of the item are the more willing to take the trouble to fill out and return the schedules. This does not mean that the amount he has of the item is the main influence in his decision to return or not to return the schedule. It means simply that the amount of the item is correlated with the factors that affect the decision. The actual amount of the item itself may be exerting no causal effect at all. The fact that such biases exist has been rather generally known ever since mail surveys were first used by statisticians. Methods for dealing with these biases have been tested from time to time. At one extreme, there have been suggestions that mail surveys be abandoned as a sampling tool and that interview-sampling methods be used exclusively. That proposal has not been universally adopted by statistical agencies because interview sampling methods are usually expensive. Furthermore, some statisticians, including the author of this article, have clung rather tenaciously to the opinion that a careful analysis of the behavior and characteristics of the respondents to mail surveys would reveal some pertinent relationships that would make it possible to estimate the extent of the bias in any survey and to make the necessary adjustments. For many problems, the application of scientific principles to the use of mail surveys would probably strengthen such surveys to the point where they would yield just as accurate results as do enumerative surveys. This is not an attempt to minimize the importance of enumerative surveys in an over-all statistical program; enumerative surveys are needed to provide the base information that must be available before mail surveys can be used scientifically. Furthermore, there will always be situations in which an enumerative survey is the most practicable method of getting data. It means, however, that a mail survey should be planned with as much attention to scientific principles as an enumerative survey. When that is done, the mail approach can be expected to yield satisfactory results in many situations in which its use has seemed undesirable. Devices that have been used to adjust the results from mail surveys include (1) enumeration by interview of a subsample of the non-respondents to the mail surveys, (2) charting of historical data from mail surveys against more accurate data obtained later by complete enumeration or similar methods, and (3) using control information that is known for both respondents and non-respondents and that is also correlated with the item to be estimated, to "true-up" the returns received by mail. All these methods, together with the directinterview type of survey itself, have been tested by statisticians of the BAE and other statistical agencies. Each seems to have its proper place in the over-all sampling program of a statistical organization. It is not the purpose of this article to give an appraisal of these methods; they are merely mentioned to provide some background for the discussion of a problem that has seemed hopeless of solution, but one that has intrigued the writer for some time.