Negative results in computer vision: A perspective

A negative result is when the outcome of an experiment or a model is not what is expected or when a hypothesis does not hold. Despite being often overlooked in the scientific community, negative results are results and they carry value. While this topic has been extensively discussed in other fields such as social sciences and biosciences, less attention has been paid to it in the computer vision community. The unique characteristics of computer vision, particularly its experimental aspect, call for a special treatment of this matter. In this paper, I will address what makes negative results important, how they should be disseminated and incentivized, and what lessons can be learned from cognitive vision research in this regard. Further, I will discuss issues such as computer vision and human vision interaction, experimental design and statistical hypothesis testing, explanatory versus predictive modeling, performance evaluation, model comparison, as well as computer vision research culture.

[1]  Leif D. Nelson,et al.  Let's Publish Fewer Papers , 2012 .

[2]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[3]  Ha Hong,et al.  Performance-optimized hierarchical models predict neural responses in higher visual cortex , 2014, Proceedings of the National Academy of Sciences.

[4]  Galit Shmueli,et al.  To Explain or To Predict? , 2010 .

[5]  Antonio Torralba,et al.  Learning visual biases from human imagination , 2014, NIPS.

[6]  R. Liang,et al.  Short-Term Effect of Ambient Temperature and the Risk of Stroke: A Systematic Review and Meta-Analysis , 2015, International journal of environmental research and public health.

[7]  Walter J. Scheirer,et al.  Using human brain activity to guide machine learning , 2017, Scientific Reports.

[8]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[9]  B. Efron,et al.  Statistical thinking for 21st century scientists , 2017, Science Advances.

[10]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[11]  Jason Yosinski,et al.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Walter J. Scheirer,et al.  Perceptual Annotation: Measuring Human Vision to Improve Computer Vision , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[14]  Rama Chellappa,et al.  Mathematical statistics and computer vision , 2012, Image Vis. Comput..

[15]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[16]  Shimon Ullman,et al.  Atoms of recognition in human and computer vision , 2016, Proceedings of the National Academy of Sciences.

[17]  G. Cumming,et al.  Researchers misunderstand confidence intervals and standard error bars. , 2005, Psychological methods.

[18]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[19]  Atsushi Okajima,et al.  Flow visualization of coaxial jet excited with varying phase differences , 2004, J. Vis..

[20]  Rob Fergus,et al.  Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks , 2015, NIPS.

[21]  Matthias Bethge,et al.  Comment on "Biologically inspired protection of deep networks from adversarial attacks" , 2017, ArXiv.

[22]  Richard I. Hartley,et al.  In Defense of the Eight-Point Algorithm , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Benjamin W Tatler,et al.  The central fixation bias in scene viewing: selecting an optimal viewing position independently of motor biases and image feature distributions. , 2007, Journal of vision.

[24]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Heinrich H. Bülthoff,et al.  Categorization of natural scenes: local vs. global information , 2006, APGV '06.

[26]  S. Plous.  The psychology of judgment and decision making , 1994 .

[27]  Frédéric Gosselin,et al.  Bubbles: a technique to reveal the use of information in recognition tasks , 2001, Vision Research.

[28]  David D. Cox,et al.  Untangling invariant object recognition , 2007, Trends in Cognitive Sciences.

[29]  C. Dunnett.  A Multiple Comparison Procedure for Comparing Several Treatments with a Control , 1955 .

[30]  Nicolas Pinto,et al.  Why is Real-World Visual Object Recognition Hard? , 2008, PLoS Comput. Biol..

[31]  Li Fei-Fei,et al.  Crowdsourcing in Computer Vision , 2016, Found. Trends Comput. Graph. Vis..

[32]  Thomas L. Dean,et al.  Neural Networks and Neuroscience-Inspired Computer Vision , 2014, Current Biology.

[33]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[34]  Donald Geman,et al.  Opinion: Science in the age of selfies , 2016, Proceedings of the National Academy of Sciences.

[35]  Robert M. Haralick,et al.  Computer vision theory: The lack thereof , 1986, Comput. Vis. Graph. Image Process..

[36]  R. Shankland.  Michelson-Morley Experiment , 1964 .

[37]  Surya Ganguli,et al.  Biologically inspired protection of deep networks from adversarial attacks , 2017, ArXiv.

[38]  T. Sterling.  Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa , 1959 .

[39]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[40]  R. VanRullen.  Perception Science in the Age of Deep Neural Networks , 2017, Front. Psychol..

[41]  Saurabh Gupta,et al.  Exploring Nearest Neighbor Approaches for Image Captioning , 2015, ArXiv.

[42]  T. Yarkoni,et al.  Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning , 2017, Perspectives on psychological science : a journal of the Association for Psychological Science.

[43]  Ali Borji,et al.  Objects do not predict fixations better than early saliency: a re-analysis of Einhauser et al.'s data. , 2013, Journal of vision.

[44]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[45]  Sinan Kalkan,et al.  Deep Hierarchies in the Primate Visual Cortex: What Can We Learn for Computer Vision? , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  J. Ioannidis.  Why Most Published Research Findings Are False , 2019, CHANCE.

[47]  Ali Borji,et al.  Human vs. Computer in Scene and Object Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[48]  Pierre Kornprobst,et al.  Bio-inspired computer vision: Towards a synergistic approach of artificial and biological vision , 2016, Comput. Vis. Image Underst..

[49]  L. Itti,et al.  Defending Yarbus: eye movements reveal observers' task. , 2014, Journal of vision.

[50]  Hod Lipson,et al.  Understanding Neural Networks Through Deep Visualization , 2015, ArXiv.

[51]  Michelle R. Greene,et al.  Reconsidering Yarbus: A failure to predict observers’ task from eye movement patterns , 2012, Vision Research.

[52]  Nikolaus Kriegeskorte,et al.  Deep neural networks: a new framework for modelling biological vision and brain information processing , 2015, bioRxiv.

[53]  O. Lodge.  The Michelson-Morley Experiment , 1925, Nature.

[54]  Jonathan Krause,et al.  Fine-Grained Crowdsourcing for Fine-Grained Recognition , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[55]  Tomaso Poggio,et al.  Representation Learning in Sensory Cortex: A Theory , 2014, IEEE Access.

[56]  A. L. Yarbus.  Eye Movements During Perception of Complex Objects , 1967 .

[57]  Denis Fize,et al.  Speed of processing in the human visual system , 1996, Nature.

[58]  Alan L. Yuille,et al.  Computer vision needs a core and foundations , 2012, Image and Vision Computing.

[59]  R. Matthews.  Storks Deliver Babies (p= 0.008) , 2000 .

[60]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[61]  BorjiAli,et al.  State-of-the-Art in Visual Attention Modeling , 2013 .

[62]  Alexei A. Efros,et al.  Unbiased look at dataset bias , 2011, CVPR 2011.

[63]  C. Lawrence Zitnick,et al.  Finding the weakest link in person detectors , 2011, CVPR 2011.

[64]  Ali Borji,et al.  Quantitative Analysis of Human-Model Agreement in Visual Saliency Modeling: A Comparative Study , 2013, IEEE Transactions on Image Processing.

[65]  Garrick Orchard,et al.  Benchmarking neuromorphic vision: lessons learnt from computer vision , 2015, Front. Neurosci..

[66]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[67]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[68]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[69]  T. Poggio,et al.  Hierarchical models of object recognition in cortex , 1999, Nature Neuroscience.

[70]  Brian A. Nosek,et al.  Recommendations for Increasing Replicability in Psychology † , 2013 .

[71]  Kunihiko Fukushima,et al.  Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Visual Pattern Recognition , 1982 .

[72]  Antonio Torralba,et al.  HOGgles: Visualizing Object Detection Features , 2013, 2013 IEEE International Conference on Computer Vision.

[73]  M. Potter.  Meaning in visual search. , 1975, Science.