Skilloscopy: Bayesian Modeling of Decision Makers' Skill

This paper proposes and demonstrates Skilloscopy as an approach to the assessment of decision makers. In an increasingly sophisticated, connected, and information-rich world, decision making is becoming both more important and more difficult. At the same time, modeling decision making on computers is becoming more feasible and of interest, partly because the information input to those decisions is increasingly on record. The aims of Skilloscopy are to rate and rank decision makers in a domain relative to each other; these aims do not include an analysis of why a decision is wrong or suboptimal, nor the modeling of the underlying cognitive process of making the decisions. In the proposed method, a decision-maker is characterized by a probability distribution of its competence in choosing among quantifiable alternatives. This probability distribution is derived by classic Bayesian inference from a combination of prior belief and evidence of the decisions. Thus, decision makers' skills may be better compared, rated, and ranked. The proposed method is applied and evaluated in the game domain of chess. A large set of games by players across a broad range of the World Chess Federation (FIDE) Elo ratings has been used to infer the distribution of players' rating directly from the moves they play rather than from game outcomes. Demonstration applications address questions frequently asked by the chess community regarding the stability of the Elo rating scale, the comparison of players of different eras and/or leagues, and controversial incidents possibly involving fraud. The method of Skilloscopy may be applied in any decision domain where the value of the decision-options can be quantified.