dotdotdot: an automated approach to quantify multiplex single molecule fluorescent in situ hybridization (smFISH) images in complex tissues

Multiplex single-molecule fluorescent in situ hybridization (smFISH) is a powerful method for validating RNA sequencing and emerging spatial transcriptomic data, but quantification remains a computational challenge. We present a framework for generating and analyzing smFISH data in complex tissues while overcoming autofluorescence and increasing multiplexing capacity. We developed dotdotdot (https://github.com/LieberInstitute/dotdotdot) as a corresponding software package to quantify RNA transcripts in single nuclei and perform differential expression analysis. We first demonstrate robustness of our platform in single mouse neurons by quantifying differential expression of activity-regulated genes. We then quantify spatial gene expression in human dorsolateral prefrontal cortex (DLPFC) using spectral imaging and dotdotdot to mask lipofuscin autofluorescence. We lastly apply machine learning to predict cell types and perform downstream cell type-specific expression analysis. In summary, we provide experimental workflows, imaging acquisition and analytic strategies for quantification and biological interpretation of smFISH data in complex tissues.