Layout-aware Subfigure Decomposition for Complex Figures in the Biomedical Literature

Published scientific figure is a valuable information resource, but often occur as composite images. The ImageCLEF meeting presented a shared evaluation in 2016 to use machine learning to split these composite figures into components automatically. We adapted an existing high-performance object detection method to analyze the substructure of published biomedical figures by developing a novel multi-branch output convolution neural network to predict irregular panel layouts and provide augmented training data to drive learning. Our system has an accuracy of 86.8% on the 2016 ImageCLEF Medical dataset and 83.1% on a new dataset derived from open access papers from the INTACT database of molecular interactions.

[1]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[4]  Rafael C. Jimenez,et al.  The IntAct molecular interaction database in 2012 , 2011, Nucleic Acids Res..

[5]  Bill Howe,et al.  Detecting and Dismantling Composite Visualizations in the Scientific Literature , 2015, ICPRAM.

[6]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[7]  Michael Krauthammer,et al.  Finding and Accessing Diagrams in Biomedical Publications , 2012, AMIA.

[8]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[10]  David J. Crandall,et al.  A Data Driven Approach for Compound Figure Separation Using Convolutional Neural Networks , 2017, 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR).

[11]  Oge Marques,et al.  Automatic separation of compound figures in scientific articles , 2016, Multimedia Tools and Applications.

[12]  Andrew W. Murray,et al.  MAD3 Encodes a Novel Component of the Spindle Checkpoint Which Interacts with Bub3p, Cdc20p, and Mad2p , 2000, The Journal of cell biology.

[13]  Eduard H. Hovy,et al.  Extracting Evidence Fragments for Distant Supervision of Molecular Interactions , 2017, SemSci@ISWC.

[14]  Henning Müller,et al.  Overview of the ImageCLEF 2016 Medical Task , 2016, CLEF.

[15]  Michael Krauthammer,et al.  Mining images in biomedical publications: Detection and analysis of gel diagrams , 2014, J. Biomed. Semant..

[16]  R. C. Johnston,et al.  MAD 3 Encodes a Novel Component of the Spindle Checkpoint which Interacts with Bub 3 p , Cdc 20 p , and Mad 2 p , 2000 .

[17]  Michael Krauthammer,et al.  Yale Image Finder (YIF): a new search engine for retrieving biomedical images , 2008, Bioinform..

[18]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[19]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[20]  Daekeun You,et al.  Image retrieval from scientific publications: Text and image content processing to separate multipanel figures , 2013, J. Assoc. Inf. Sci. Technol..