Performance metrics for an application-driven selection and optimization of psychophysical sampling procedures

When estimating psychometric functions with sampling procedures, psychophysical assessments should be precise and accurate while being as efficient as possible to reduce assessment duration. The estimation performance of sampling procedures is commonly evaluated in computer simulations for single psychometric functions and reported using metrics as a function of number of trials. However, the estimation performance of a sampling procedure may vary for different psychometric functions. Therefore, the results of these type of evaluations may not be generalizable to a heterogeneous population of interest. In addition, the maximum number of trials is often imposed by time restrictions, especially in clinical applications, making trial-based metrics suboptimal. Hence, the benefit of these simulations to select and tune an ideal sampling procedure for a specific application is limited. We suggest to evaluate the estimation performance of sampling procedures in simulations covering the entire range of psychometric functions found in a population of interest, and propose a comprehensive set of performance metrics for a detailed analysis. To illustrate the information gained from these metrics in an application example, six sampling procedures were evaluated in a computer simulation based on prior knowledge on the population distribution and requirements from proprioceptive assessments. The metrics revealed limitations of the sampling procedures, such as inhomogeneous or systematically decreasing performance depending on the psychometric functions, which can inform the tuning process of a sampling procedure. More advanced metrics allowed directly comparing overall performances of different sampling procedures and select the best-suited sampling procedure for the example application. The proposed analysis metrics can be used for any sampling procedure and the estimation of any parameter of a psychometric function, independent of the shape of the psychometric function and of how such a parameter was estimated. This framework should help to accelerate the development process of psychophysical assessments.

[1]  C D Creelman,et al.  PEST reduces bias in forced choice psychophysics. , 1983, The Journal of the Acoustical Society of America.

[2]  Olivier Lambercy,et al.  Experimental Validation of a Rapid, Adaptive Robotic Assessment of the MCP Joint Angle Difference Threshold , 2014, EuroHaptics.

[3]  D. M. Green,et al.  A maximum-likelihood method for estimating thresholds in a yes-no task. , 1993, The Journal of the Acoustical Society of America.

[4]  A. Watson,et al.  The method of constant stimuli is inefficient , 1990, Perception & psychophysics.

[5]  Hong Z. Tan,et al.  Discrimination and identification of finger joint-angle position using active motion , 2007, TAP.

[6]  Nicolaas Prins,et al.  The psychometric function: the lapse rate revisited. , 2012, Journal of vision.

[7]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[8]  F A Wichmann,et al.  Ning for Helpful Comments and Suggestions. This Paper Benefited Con- Siderably from Conscientious Peer Review, and We Thank Our Reviewers the Psychometric Function: I. Fitting, Sampling, and Goodness of Fit , 2001 .

[9]  H. Levitt.  Transformed up-down methods in psychoacoustics. , 1971, The Journal of the Acoustical Society of America.

[10]  H Strasburger,et al.  Converting between measures of slope of the psychometric function , 2001, Perception & psychophysics.

[11]  S. Klein,et al.  Measuring, estimating, and understanding the psychometric function: A commentary , 2001, Perception & psychophysics.

[12]  S. McKee,et al.  Statistical properties of forced-choice psychometric functions: Implications of probit analysis , 1985, Perception & psychophysics.

[13]  N. Prins.  The psi-marginal adaptive method: How to give nuisance parameters the attention they deserve (no more, no less). , 2013, Journal of vision.

[14]  L. Carey,et al.  Somatosensory assessment and treatment after stroke: An evidence-practice gap. , 2015, Australian occupational therapy journal.

[15]  J. Maunsell,et al.  When Attention Wanders: How Uncontrolled Fluctuations in Attention Affect Performance , 2011, The Journal of Neuroscience.

[16]  Neil A. Macmillan,et al.  Detection Theory: A User's Guide , 1991 .

[17]  C. Tyler,et al.  Bayesian adaptive estimation of psychometric slope and threshold , 1999, Vision Research.

[18]  J. L. Hall.  Hybrid adaptive procedure for estimation of psychometric functions. , 1980, The Journal of the Acoustical Society of America.

[19]  A Pentland,et al.  Maximum likelihood estimation: The best PEST , 1980, Perception & psychophysics.

[20]  M. García-Pérez,et al.  Sampling Plans for Fitting the Psychometric Function , 2005, The Spanish Journal of Psychology.

[21]  Claude Ghez,et al.  A robotic test of proprioception within the hemiparetic arm post-stroke , 2014, Journal of NeuroEngineering and Rehabilitation.

[22]  J. L. Hall,et al.  A procedure for detecting variability of psychophysical thresholds. , 1983, The Journal of the Acoustical Society of America.

[23]  K. Kamibayashi.  Motor Control and Learning: A Behavioral Emphasis, 6th Edition , 2019, Medicine & Science in Sports & Exercise.

[24]  Luca Faes,et al.  Small-sample characterization of stochastic approximation staircases in forced-choice adaptive threshold estimation , 2007, Perception & psychophysics.

[25]  R. Madigan,et al.  Maximum-likelihood psychometric procedures in two-alternative forced-choice: Evaluation and recommendations , 1987, Perception & psychophysics.

[26]  Olivier Lambercy,et al.  Design of a robotic device for assessment and rehabilitation of hand sensory function , 2011, 2011 IEEE International Conference on Rehabilitation Robotics.

[27]  Pamela W. Duncan,et al.  Post-stroke rehabilitation: Assessment, referral, and patient management: Quick reference guide for clinicians , 1996 .

[28]  Olivier Lambercy,et al.  Reliability, validity, and clinical feasibility of a rapid and objective assessment of post-stroke deficits in hand proprioception , 2018, Journal of NeuroEngineering and Rehabilitation.

[29]  Nadina B. Lincoln,et al.  The unreliability of sensory assessments , 1991 .

[30]  A. Turpin,et al.  Identifying steep psychometric function slope quickly in clinical applications , 2010, Vision Research.

[31]  Richard A. Tyrrell,et al.  A rapid technique to assess the resting states of the eyes and other threshold phenomena: The Modified Binary Search (MOBS) , 1988 .

[32]  Jürgen Konczak,et al.  Assessing Proprioceptive Function: Evaluating Joint Position Matching Methods Against Psychophysical Thresholds , 2013, Physical Therapy.

[33]  Felix Wichmann,et al.  The psychometric function: II. Bootstrap-based confidence intervals and sampling , 2001, Perception & psychophysics.

[34]  C S Watson,et al.  Time course of auditory perceptual learning. , 1980, The Annals of otology, rhinology & laryngology. Supplement.

[35]  Olivier Lambercy,et al.  Algorithm for improving psychophysical threshold estimates by detecting sustained inattention in experiments using PEST , 2018, Attention, Perception, & Psychophysics.

[36]  Hong Z. Tan,et al.  HUMAN FACTORS FOR THE DESIGN OF FORCE-REFLECTING HAPTIC INTERFACES , 1994 .

[37]  M R Leek,et al.  An interleaved tracking procedure to monitor unstable psychometric functions. , 1991, The Journal of the Acoustical Society of America.

[38]  W A Simpson.  The step method: A new adaptive psychophysical procedure , 1989, Perception & psychophysics.

[39]  John M. Findlay,et al.  Estimates on probability functions: A more virulent PEST , 1978 .

[40]  H. Strasburger,et al.  Fitting the psychometric function , 1999, Perception & psychophysics.

[41]  L. Hedman,et al.  Sensory Dysfunction Following Stroke: Incidence, Significance, Examination, and Intervention , 2008, Topics in stroke rehabilitation.

[42]  P. King-Smith,et al.  Efficient and unbiased modifications of the QUEST threshold method: Theory, simulations, experimental evaluation and practical implementation , 1994, Vision Research.

[43]  J. O'Regan,et al.  Estimating psychometric functions in forced-choice situations: Significant biases found in threshold and slope estimations when small samples are used , 1989, Perception & psychophysics.

[44]  Ingo Fründ,et al.  Inference for psychometric functions in the presence of nonstationary behavior. , 2011, Journal of vision.

[45]  Olivier Lambercy,et al.  Age-based model for metacarpophalangeal joint proprioception in elderly , 2017, Clinical interventions in aging.

[46]  Y. Matsuoka,et al.  Perceptual limits for a robotic rehabilitation environment using visual feedback distortion , 2005, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[47]  A. Watson,et al.  Quest: A Bayesian adaptive psychometric method , 1983, Perception & psychophysics.

[48]  M. Leek.  Adaptive procedures in psychophysical research , 2001, Perception & psychophysics.

[49]  B. Treutwein.  Adaptive psychophysical procedures , 1995, Vision Research.

[50]  J. Konczak,et al.  Robot-Aided Assessment of Wrist Proprioception , 2015, Front. Hum. Neurosci..

[51]  M. M. Taylor.  On the efficiency of psychophysical measurement. , 1971, The Journal of the Acoustical Society of America.

[52]  M. M. Taylor,et al.  PEST: Efficient Estimates on Probability Functions , 1967 .

[53]  C Kaernbach,et al.  Simple adaptive testing with the weighted up-down method , 1991, Perception & psychophysics.

[54]  H. Kesten.  Accelerated Stochastic Approximation , 1958 .

[55]  Alexander M. Mood,et al.  A Method for Obtaining and Analyzing Sensitivity Data , 1948 .

[56]  T. Cornsweet,et al.  The staircrase-method in psychophysics. , 1962, The American journal of psychology.

[57]  P. Veltink,et al.  Observation of time-dependent psychophysical functions and accounting for threshold drifts , 2015, Attention, perception & psychophysics.