Does My Device Leak Information? An a priori Statistical Power Analysis of Leakage Detection Tests

The development of a leakage detection testing methodology for the side-channel resistance of cryptographic devices is an issue that has received recent focus from standardisation bodies such as NIST. Statistical techniques such as hypothesis and significance testing appear to be ideally suited for this purpose. In this work we evaluate the candidacy of three such detection tests: a t-test proposed by Cryptography Research Inc., and two mutual information-based tests, one in which data is treated as continuous and one as discrete. Our evaluation investigates three particular areas: statistical power, the effectiveness of multiplicity corrections, and computational complexity. To facilitate a fair comparison we conduct a novel a priori statistical power analysis of the three tests in the context of side-channel analysis, finding surprisingly that the continuous mutual information and t-tests exhibit similar levels of power. We also show how the inherently parallel nature of the continuous mutual information test can be leveraged to reduce a large computational cost to insignificant levels. To complement the a priori statistical power analysis we include two real-world case studies of the tests applied to software and hardware implementations of the AES.