Gene

Biology

Messenger RNA

Fluorescence in situ hybridization

Computer science

In situ

Computational biology

Geology

Immunostaining

Image segmentation

Spatial analysis

Immunology

Genetics

Transcriptome

Organic chemistry

Segmentation

Pattern recognition (psychology)

In situ hybridization

Artificial intelligence

Remote sensing

Chemistry

Immunohistochemistry

Gene expression

Chromosome

Computer vision

Nature Biotechnology

Single-molecule spatial transcriptomics protocols based on in situ sequencing or multiplexed RNA fluorescent hybridization can reveal detailed tissue organization. However, distinguishing the boundaries of individual cells in such data is challenging and can hamper downstream analysis. Current methods generally approximate cells positions using nuclei stains. We describe a segmentation method, Baysor, that optimizes two-dimensional (2D) or three-dimensional (3D) cell boundaries considering joint likelihood of transcriptional composition and cell morphology. While Baysor can take into account segmentation based on co-stains, it can also perform segmentation based on the detected transcripts alone. To evaluate performance, we extend multiplexed error-robust fluorescence in situ hybridization (MERFISH) to incorporate immunostaining of cell boundaries. Using this and other benchmarks, we show that Baysor segmentation can, in some cases, nearly double the number of cells compared to existing tools while reducing segmentation artifacts. We demonstrate that Baysor performs well on data acquired using five different protocols, making it a useful general tool for analysis of imaging-based spatial transcriptomics. Baysor enables cell segmentation based on transcripts detected by multiplexed FISH or in situ sequencing.

Cell segmentation in imaging-based spatial transcriptomics