Fig. 2

a Overview of GeneSegNet framework. GeneSegNet makes a joint use of gene spatial coordinates and imaging information for cell segmentation, and is recursively learned by alternating between the optimization of network parameters and estimation of training labels for noise-tolerant training. The outputs of GeneSegNet include a confidence map \(\hat{M} \in \left[ 0, 1 \right] ^{w \times h}\) that stores the probability of each pixel being inside cell regions, a center map \(\hat{C} \in \left[ 0, 1 \right] ^{w \times h}\) that gives the likelihood of each pixel being a cell center, and a two-channel offset map \(\hat{V} \in \mathbb {R}^{2 \times w \times h}\) indicating the offset vector of each pixel to the center of its corresponding cell instance. Combining the outputs together, GeneSegNet can recover the precise boundary of each individual cell. b GeneSegNet is built upon the general U-Net network architecture, where an encoder progressive downsamples the inputs for more expressive feature extraction, and a decoder upsamples the feature maps in a mirror-symmetric fashion for fine-grained cell segmentation