Do Neural Networks for Segmentation Understand Insideness?

The insideness problem is an image segmentation modality that consists of determining which pixels are inside and outside a region. Deep Neural Networks (DNNs) excel in segmentation benchmarks, but it is unclear that they have the ability to solve the insideness problem as it requires evaluating longrange spatial dependencies. In this paper, the insideness problem is analysed in isolation, without texture or semantic cues, such that other aspects of segmentation do not interfere in the analysis. We demonstrate that DNNs for segmentation with few units have sufficient complexity to solve insideness for any curve. Yet, such DNNs have severe problems to learn general solutions. Only recurrent networks trained with small images learn solutions that generalize well to almost any curve. Recurrent networks can decompose the evaluation of long-range dependencies into a sequence of local operations, and learning with small images alleviates the common difficulties of training recurrent networks with a large number of unrolling steps. This material is based upon work supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216. Do Neural Networks for Segmentation Understand Insideness? Kimberly Villalobos∗,1, Vilim Štih∗,1,2, Amineh Ahmadinejad∗,1, Shobhita Sundaram, Jamell Dozier, Andrew Francl, Frederico Azevedo, Tomotake Sasaki†,3,1, Xavier Boix†,1,• ∗ and † indicate equal contribution 1 Center for Brains, Minds and Machines, MIT (USA) 2 Max Planck Institute of Neurobiology (Germany) 3 Fujitsu Laboratories Ltd. (Japan) • Correspondence to xboix@mit.edu