Pitfalls of Shoulder Surfing Studies

We review empirical studies that evaluate the resilience of various PIN entry methods against human shoulder surfers. Conducting such studies is challenging because adversaries are not available for study and must be simulated in one way or another. We were interested to find out whether there is a common standard how these experiments are designed and reported. In the course of our research we noticed that subtle design decisions might have a crucial effect on the validity and the interpretation of the outcomes. Getting these details right is particularly important if the number of participants or trials is relatively low. One example is the decision to let simulated adversaries enter their guesses using the method under study. If the method produces input errors then correct guesses may not be counted as such, which leads to an underestimation of risk. We noticed several issues of this kind and distilled a set of recommendations that we believe should be followed to assure that studies of this kind are comparable and that their results can be interpreted well.