Vision‐based and marker‐less surgical tool detection and tracking: a review of the literature

&NA; In recent years, tremendous progress has been made in surgical practice for example with Minimally Invasive Surgery (MIS). To overcome challenges coming from deported eye‐to‐hand manipulation, robotic and computer‐assisted systems have been developed. Having real‐time knowledge of the pose of surgical tools with respect to the surgical camera and underlying anatomy is a key ingredient for such systems. In this paper, we present a review of the literature dealing with vision‐based and marker‐less surgical tool detection. This paper includes three primary contributions: (1) identification and analysis of data‐sets used for developing and testing detection algorithms, (2) in‐depth comparison of surgical tool detection methods from the feature extraction process to the model learning strategy and highlight existing shortcomings, and (3) analysis of validation techniques employed to obtain detection performance results and establish comparison between surgical tool detectors. The papers included in the review were selected through PubMed and Google Scholar searches using the keywords: “surgical tool detection”, “surgical tool tracking”, “surgical instrument detection” and “surgical instrument tracking” limiting results to the year range 2000 2015. Our study shows that despite significant progress over the years, the lack of established surgical tool data‐sets, and reference format for performance assessment and method ranking is preventing faster improvement. HighlightsIn‐depth state‐of‐the‐art review of surgical tool detection from 24 recent papers.Lack of a standard format regarding datasets employed and corresponding annotations.Comprehensive highlighting of advantages and disadvantages of existing methods.Limited consensus on a common reference format within validation methodologies. Graphical abstract Figure. No caption available.