A weakly supervised activity recognition framework for real-time synthetic biology laboratory assistance

We describe the design of a hybrid system -- a combination of a Dynamic Graphical Model (DGM) with a Deep Neural Network (DNN) -- to identify activities performed during synthetic biology experiments. The purpose is to provide real-time feedback to experimenters, thus helping to reduce human errors and improve experimental reproducibility. The data consists of unlabeled videos of recorded experiments and "weakly supervised" information (i.e., "theoretical" and asynchronous knowledge of sets of high level activity sequences in the experiment) used to train the system. Multiple activity sequences are modeled using a trellis, and deep features are extracted from video images. Model performance is accessed using real-time online statistical inference. The trellis incorporates variations during experiment execution, making our model very general and capable of high performance.

[1]  Yunqian Ma,et al.  Hierarchical Model-Based Activity Recognition With Automatic Low-Level State Discovery , 2007, J. Multim..

[2]  Alan Julian Izenman,et al.  Modern Multivariate Statistical Techniques , 2008 .

[3]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[4]  J. Collins,et al.  Construction of a genetic toggle switch in Escherichia coli , 2000, Nature.

[5]  Léon Bottou,et al.  Stochastic Gradient Descent Tricks , 2012, Neural Networks: Tricks of the Trade.

[6]  Karen M Polizzi.  What is synthetic biology? , 2013, Methods in molecular biology.

[7]  Andrea Vedaldi,et al.  Understanding deep image representations by inverting them , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Jeff A. Bilmes,et al.  On Virtual Evidence and Soft Evidence in Bayesian Networks , 2004 .

[9]  Philipp Scholl,et al.  Wearables in the wet lab: a laboratory system for capturing and guiding experiments , 2015, UbiComp.

[10]  Oren Etzioni,et al.  A Sound and Fast Goal Recognizer , 1995, IJCAI.

[11]  Atsuto Maki,et al.  From generic to specific deep representations for visual recognition , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[12]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[13]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[14]  B. Nadler,et al.  Diffusion maps, spectral clustering and reaction coordinates of dynamical systems , 2005, math/0503445.

[15]  Rémi Ronfard,et al.  A survey of vision-based methods for action representation, segmentation and recognition , 2011, Comput. Vis. Image Underst..

[16]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[17]  Henry A. Kautz.  A formal theory of plan recognition , 1987 .

[18]  D. G. Gibson,et al.  Enzymatic assembly of DNA molecules up to several hundred kilobases , 2009, Nature Methods.

[19]  A. Laub,et al.  The singular value decomposition: Its computation and some applications , 1980 .

[20]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[21]  Trevor Darrell,et al.  Long-term recurrent convolutional networks for visual recognition and description , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Philipp Scholl,et al.  RFID-based compound identification in wet laboratories with google glass , 2015, iWOAR.

[23]  Erika Check Hayden,et al.  Synthetic biologists seek standards for nascent field , 2015, Nature.

[24]  Rafael Silva-Rocha,et al.  Mining logic gates in prokaryotic transcriptional regulation networks , 2008, FEBS letters.

[25]  Geoffrey Zweig,et al.  The graphical models toolkit: An open source software system for speech and time-series processing , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[26]  Jenq-Neng Hwang,et al.  A Review on Video-Based Human Activity Recognition , 2013, Comput..

[27]  Pedro J. Moreno,et al.  A factor automaton approach for the forced alignment of long speech recordings , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[28]  J. Liao,et al.  A synthetic gene–metabolic oscillator , 2005, Nature.

[29]  Gwenn Englebienne,et al.  Hierarchical Activity Recognition Using Automatically Clustered Actions , 2011, AmI.

[30]  LeCunYann,et al.  Learning Hierarchical Features for Scene Labeling , 2013 .

[31]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[32]  Camille Couprie,et al.  Learning Hierarchical Features for Scene Labeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[34]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Robert P. Goldman,et al.  A Bayesian Model of Plan Recognition , 1993, Artif. Intell..

[36]  Irene Kuhn,et al.  Sorting out the FACS: a devil in the details. , 2014, Cell reports.

[37]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[38]  Gwenn Englebienne,et al.  Accurate activity recognition in a home setting , 2008, UbiComp.

[39]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[40]  Erika Check Hayden.  Synthetic biology called to order. , 2015 .

[41]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[42]  Hervé Bourlard,et al.  Connectionist Speech Recognition: A Hybrid Approach , 1993 .

[43]  D. Stirling,et al.  A short history of the polymerase chain reaction. , 2003, Methods in molecular biology.

[44]  Stéphane Lafon,et al.  Diffusion maps , 2006 .

[45]  Xiaogang Wang,et al.  Deep Learning Face Representation from Predicting 10,000 Classes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Judea Pearl,et al.  Chapter 2 – BAYESIAN INFERENCE , 1988 .

[47]  Matti Pietikäinen,et al.  Dynamic texture and scene classification by transferring deep image features , 2015, Neurocomputing.

[48]  Svetha Venkatesh,et al.  Recognition of human activity through hierarchical stochastic learning , 2003, Proceedings of the First IEEE International Conference on Pervasive Computing and Communications, 2003. (PerCom 2003)..

[49]  Robert H. Riffenburgh,et al.  Linear Discriminant Analysis , 1960 .

[50]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[51]  Gaetano Borriello,et al.  Microbiology Tray and Pipette Tracking as a Proactive Tangible User Interface , 2004, Pervasive.

[52]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[53]  Trevor Darrell,et al.  DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition , 2013, ICML.