Minimalistic Image Signal Processing for Deep Learning Applications

In-sensor energy-efficient deep learning accelerators have the potential to enable the use of deep neural networks in embedded vision applications. However, their negative impact on accuracy has been severely underestimated. The inference pipeline used in prior in-sensor deep learning accelerators bypasses the image signal processor (ISP), thereby disrupting the conventional vision pipeline and undermining accuracy of machine learning algorithms trained on conventional, post-ISP datasets. For example, the detection accuracy of an off-the-shelf Faster RCNN algorithm in a vehicle detection scenario reduces by 60%. To make in-sensor accelerators practical, we describe energy-efficient operations that yield most of the benefits of an ISP and reduce covariate shift between the training (ISP processed images) and target (RAW images) distributions. For the vehicle detection problem, our approach improves accuracy by 25–60%. Relative to the conventional ISP pipeline, energy consumption and response time improve by 30% and 34%, respectively.