Fontnet: On-Device Font Understanding and Prediction Pipeline

Fonts are one of the most basic and core design concepts. Numerous use cases can benefit from an in depth understanding of Fonts such as Text Customization which can change text in an image while maintaining the Font attributes like style, color, size. Currently, Text recognition solutions can group recognized text based on line breaks or paragraph breaks, if the Font attributes are known multiple text blocks can be combined based on context in a meaningful manner. In this paper, we propose two engines: Font Detection Engine, which identifies the font style, color and size attributes of text in an image and a Font Prediction Engine, which predicts similar fonts for a query font. Major contributions of this paper are three-fold: First, we developed a novel CNN architecture for identifying font style of text in images. Second, we designed a novel algorithm for predicting similar fonts for a given query font. Third, we have optimized and deployed the entire engine On-Device which ensures privacy and improves latency in real time applications such as instant messaging. We achieve a worst case On-Device inference time of 30ms and a model size of 4.5MB for both the engines.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Alexandru Telea,et al.  An Image Inpainting Technique Based on the Fast Marching Method , 2004, J. Graphics, GPU, & Game Tools.

[3]  Yifan Chang.  Chinese Font Recognition Based on Convolution Neural Network , 2018 .

[4]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[5]  Yizhi Wang,et al.  Font Recognition in Natural Images via Transfer Learning , 2018, MMM.

[6]  Aaron Hertzmann,et al.  Exploratory font selection using crowdsourced attributes , 2014, ACM Trans. Graph..

[7]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Neural Networks , 2013 .

[8]  Thomas S. Huang,et al.  DeepFont: Identify Your Font from An Image , 2015, ACM Multimedia.

[9]  Gerard de Melo,et al.  Predicting Semantic Signatures of Fonts , 2018, 2018 IEEE 12th International Conference on Semantic Computing (ICSC).

[10]  Author.  $article.title , 2002, Nature.

[11]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[12]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[13]  Rynson W. H. Lau,et al.  Modeling Fonts in Context: Font Prediction on Web Designs , 2018, Comput. Graph. Forum.

[14]  Tony X. Han,et al.  Large-Scale Visual Font Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.