Execution Time Modeling for CNN Inference on Embedded GPUs

Machine learning is one of the most cutting edge methods in computer vision. Convolutional Neural Networks (CNN) in particular are widely used in edge computing based applications such as autonomous driving for image recognition or object tracking. Different constraints exist in this application area such as real-time, energy consumption, memory resources, etc. Choosing the optimal CNN for each GPU at hand is really hard to do, while maintaining high levels of accuracy and performance. This makes prior knowledge about the execution time a necessary prerequisite information before the final deployment of the CNN on the edge GPU platform. In this paper, we compare 5 execution time prediction models on a large set of CNNs-based applications. The tested predictors use machine learning regression approach. The proposed methodology is based on the utilization of high level CNN features. At the opposite of state-of-the-art approaches, no implementation or profiling on the hardware is required. A Mean Absolute Percentage Error (MAPE) of 5% using Support Vector Regression and Artificial Neural Networks has been obtained in the experiments. Our comparison shows the efficiency of these models to rapidly explore a large space of CNN models or Hardware configurations.