From One-Trick Ponies to All-Rounders: On-Demand Learning for Image Restoration

While machine learning approaches to image restoration offer great promise, current methods risk training “onetrick ponies” that perform well only for image corruption of a particular level of difficulty—such as a certain level of noise or blur. First, we examine the weakness of a one-trick pony model and demonstrate that training general models to handle arbitrary levels of corruption is indeed non-trivial. Then, we propose an on-demand learning algorithm for training image restoration models with deep convolutional neural networks. The main idea is to exploit a feedback mechanism to self-generate training instances where they are needed most, thereby learning models that can generalize across difficulty levels. On four restoration tasks—image inpainting, pixel interpolation, image deblurring, and image denoising—and three diverse datasets, our approach consistently outperforms both the status quo training procedure and curriculum learning alternatives.