Seeing the Arrow of Time

We explore whether we can observe Time's Arrow in a temporal sequence - is it possible to tell whether a video is running forwards or backwards? We investigate this somewhat philosophical question using computer vision and machine learning techniques. We explore three methods by which we might detect Time's Arrow in video sequences, based on distinct ways in which motion in video sequences might be asymmetric in time. We demonstrate good video forwards/backwards classification results on a selection of YouTube video clips, and on natively-captured sequences (with no temporally-dependent video compression), and examine what motions the models have learned that help discriminate forwards from backwards time.

[1]  O. Penrose.  The Direction of Time , 1962 .

[2]  K. Fieandt,et al.  The perceptual world , 1977 .

[3]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[4]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..

[5]  C. Tomasi.  Detection and Tracking of Point Features , 1991 .

[6]  J. Bartlett,et al.  Inversion and processing of component and spatial–relational information in faces. , 1996 .

[7]  J. Bartlett,et al.  Inversion and processing of component and spatial-relational information in faces. , 1996, Journal of experimental psychology. Human perception and performance.

[8]  H. Price.  Time's arrow and Archimedes' point new directions for the physics of time , 1997 .

[9]  Martin Greiner,et al.  Wavelets , 2018, Complex..

[10]  R Kemp,et al.  Perception and Recognition of Normal and Negative Faces: The Role of Shape from Shading and Pigmentation Cues , 1996, Perception.

[11]  David J. Field,et al.  Wavelets, vision and the statistics of natural scenes , 1999, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[12]  Guillermo Sapiro,et al.  A Variational Model for Filling-In Gray Level and Color Images , 2001, ICCV.

[13]  R. Aslin,et al.  Statistical learning of higher-order temporal structure from visual shape sequences. , 2002, Journal of experimental psychology. Learning, memory, and cognition.

[14]  E. Shechtman,et al.  Transactions on Pattern Analysis and Machine Intelligence 1 Space-time Video Completion Draft Transactions on Pattern Analysis and Machine Intelligence 2 , 2022 .

[15]  Eero P. Simoncelli,et al.  A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients , 2000, International Journal of Computer Vision.

[16]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[17]  Michael J. Black,et al.  Fields of Experts: a framework for learning image priors , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[18]  Cordelia Schmid,et al.  Learning realistic human actions from movies , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Tuomas K. Pernu,et al.  In the light of time , 2009, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[20]  Bernhard Schölkopf,et al.  Detecting the direction of causal time series , 2009, ICML '09.

[21]  Andrea Vedaldi,et al.  Vlfeat: an open and portable library of computer vision algorithms , 2010, ACM Multimedia.

[22]  Cordelia Schmid,et al.  Action recognition by dense trajectories , 2011, CVPR 2011.

[23]  Yair Weiss,et al.  From learning models of natural image patches to whole image restoration , 2011, 2011 International Conference on Computer Vision.

[24]  Richard P. Wildes,et al.  Dynamic scene understanding: The role of orientation features in space and time in scene classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Bernhard Schölkopf,et al.  On causal and anticausal learning , 2012, ICML.

[26]  Cordelia Schmid,et al.  Temporal Localization of Actions with Actoms. , 2013, IEEE transactions on pattern analysis and machine intelligence.

[27]  Shai Avidan,et al.  Photo Sequencing , 2014, International Journal of Computer Vision.