A Joint Deep Neural Network and Evidence Accumulation Modeling Approach to Human Decision-Making with Naturalistic Images

Evidence accumulation models (EAM) have proven to be an invaluable tool in probing the dynamical properties of decisions over recent decades. However, much of this literature has studied decisions utilizing simple stimuli where the experimenter has perfect knowledge and control over stimulus properties. Here, we develop and test a new method for studying decisions involving naturalistic stimuli (medical images in this case) where the experimenter has neither perfect knowledge nor control of the stimuli properties. The central challenge in studying such decisions is to extract useful representations of images that can be associated with accumulation or drift rates in EAMs. Here, we couple a deep convolutional neural network (CNN) with the diffusion decision model (DDM) to study how expert pathologists and novices make decisions involving the classification of digital images of blood cells as either normal (non-blast) or cancerous (blast). In our approach, the CNN is the basis of a function that translates each image into a drift rate for use in the DDM. Results of fitting the joint CNN-DDM model to choice and response time data demonstrates that (1) both novices and experts demonstrated substantial speed accuracy tradeoffs, (2) both were susceptible to biases introduced by the presentation of pre-stimulus probabilistic cues, and (3) experts were more adept at extracting useful information from images than novices. These results demonstrate that this is a fruitful approach to studying decisions involving complex stimuli that will open new avenues for studying questions not possible with existing methods. Furthermore, this approach is technically feasible and has the potential to be translated into other domains of decision-making research.