Design, analysis, and inference for studies comparing thematic accuracy of classified remotely sensed data: a special case of map comparison

Assessing thematic map accuracy is a special type of map comparison that is frequently applied to remote sensing classification problems. For map comparisons in the accuracy assessment setting, one map represents the classified output and the other map represents the true or “reference” condition. Several articles in this special issue describe state-of-the-art map comparison analysis tools that could serve to quantify accuracy of a single map. However, accuracy assessment objectives generally extend beyond describing accuracy of a single map to comparing accuracy of several maps. Consequently, interest focuses on comparing map comparison measures when these measures are used to represent accuracy. The virtual workshop emphasizes the analysis component of map comparisons, but it is also important to examine the underlying study designs generating the data input into these analyses. The study designs for accuracy comparisons implemented in remote sensing practice often investigate only a single test site, thus limiting our ability to generalize the results of these accuracy comparisons. Map accuracy comparison studies can be designed to provide stronger generalizations by incorporating experimental design principles such as replication and blocking, and identifying an experimental unit appropriate for the application. It is also important to recognize the role of statistical hypothesis testing and inference for different objectives that motivate map accuracy comparisons. Deciding which of two maps to use for a particular site can be addressed by enumerative inference and does not require hypothesis testing. For the objective of a more general comparison of classification procedures, analytic inference is appropriate and hypothesis testing plays a more prominent role.