Collecting and Analyzing Expert Group Judgment Data

Abstract : This paper presents methods for collecting and analyzing judgement from groups of experts. It addresses issues associated with resolving the procedural and administrative problems involved in selecting a panel of experts, in eliciting informed judgments about the degree of technological advance on relevant projects, and in designing a survey questionnaire for measuring those judgments. Three methods of multivariate analysis are described for quantifying and analyzing group judgment data collected from a panel of experts. Those of the methods that are known in earlier literature include multidimensional scaling of individual differences, and subjective probability procedures, including the Bayesian approach. A new procedure, which was developed specifically for this application, involves the use of simultaneous equation system models in which the response (dependent) variables are categorical and unordered. (Author)