Medical Image Segmentation
Segmenting organs and abnormalities in medical images.
Medical image segmentation delineates anatomical structures and pathologies in 3D medical scans — critical for treatment planning, tumor measurement, and surgical guidance. nnU-Net remains the dominant auto-configuring baseline, while SAM-Med2D and Med-SA bring foundation model capabilities to medical segmentation.
History
U-Net (Ronneberger et al.) establishes the encoder-decoder architecture for biomedical segmentation
V-Net extends U-Net to 3D volumetric segmentation for CT/MRI
nnU-Net auto-configures preprocessing, architecture, and training for any medical segmentation task
Attention U-Net adds attention gates for focusing on relevant anatomical regions
UNETR applies vision transformers to 3D medical image segmentation
nnU-Net wins multiple Medical Segmentation Decathlon challenges
SAM (Segment Anything) adapted for medical images — SAM-Med2D and MedSAM
STU-Net scales 3D medical segmentation to 1.4B parameters
TotalSegmentator provides 117-class whole-body CT segmentation
Foundation segmentation models enable zero-shot organ and tumor delineation
How Medical Image Segmentation Works
Image Acquisition
3D medical volumes (CT, MRI, PET) or 2D images (X-ray, ultrasound, pathology slides) are acquired and standardized.
Preprocessing
nnU-Net-style preprocessing: intensity normalization, resampling to target spacing, and cropping to the region of interest.
Encoder Path
A CNN (ResNet blocks) or ViT encoder compresses the image into a hierarchy of feature maps at decreasing resolutions.
Decoder Path
The decoder upsamples features back to input resolution, with skip connections from the encoder preserving fine-grained spatial detail.
Segmentation Output
A pixel/voxel-wise classification produces a segmentation mask, with each pixel labeled as background or one of the target structures.
Current Landscape
Medical image segmentation in 2025 is dominated by nnU-Net — the auto-configuring framework that consistently achieves top results across diverse tasks without manual tuning. Foundation models (MedSAM, SAM-Med2D) are enabling interactive and few-shot segmentation, reducing the annotation burden. TotalSegmentator demonstrates that comprehensive whole-body segmentation is now production-ready. The field is mature for common structures (organs, major vessels) but still challenging for small lesions, rare pathologies, and domains with limited training data.
Key Challenges
Annotation cost — expert radiologist/pathologist annotation of 3D volumes is extremely expensive ($50-200 per scan)
Class imbalance — lesions and tumors are tiny relative to background, requiring focal loss or oversampling
3D memory constraints — processing full-resolution 3D volumes requires patch-based training with careful aggregation
Domain gaps — different scanners, protocols, and patient populations cause segmentation failures on new data
Evaluation metrics — Dice score alone is insufficient; surface distance and volumetric accuracy matter for clinical use
Quick Recommendations
Any new segmentation task
nnU-Net
Auto-configuring, consistently wins or ties SOTA on new datasets with zero manual tuning
Interactive segmentation
MedSAM / SAM-Med2D
Click-based prompting for quick annotation and novel structure segmentation
Whole-body CT
TotalSegmentator
117 anatomical structures in one pass, production-ready
Research with transformers
SwinUNETR / UNETR
Best transformer-based architectures for 3D medical volumes
What's Next
The frontier is universal medical segmentation — models that can segment any anatomical structure or pathology from any imaging modality with minimal or zero task-specific training. Expect foundation models pretrained on massive multi-institutional datasets, federated learning to access data across hospitals, and real-time segmentation for surgical guidance.
Benchmarks & SOTA
Synapse Multi-Organ CT
Synapse Multi-Organ Abdominal CT Segmentation
18 abdominal CT scans with 13 organ labels. Most widely used multi-organ CT segmentation benchmark. MICCAI 2015 Multi-Atlas Abdomen Labeling Challenge.
State of the Art
SegMamba
Xing et al.
86.45
mean-dsc
ACDC
Automated Cardiac Diagnosis Challenge
150 cardiac MRI cases segmenting right ventricle, myocardium, and left ventricle. MICCAI 2017 challenge.
State of the Art
MedNeXt-L
German Cancer Research Center (DKFZ)
92.65
mean-dsc
BTCV
Beyond The Cranial Vault Multi-Organ CT Segmentation
30 abdominal CT scans with 13 organ annotations. MICCAI 2015 challenge hosted on Synapse platform.
State of the Art
STU-Net-H
Ziyan Huang et al.
85.38
mean-dsc
BraTS 2023
Brain Tumor Segmentation Challenge 2023
1,251 multi-institutional MRI cases for brain tumor segmentation. Segments whole tumor (WT), tumor core (TC), and enhancing tumor (ET). Annual MICCAI challenge.
State of the Art
MedNeXt-L
German Cancer Research Center (DKFZ)
0.896
mean-dice-wt-tc-et
Related Tasks
Something wrong or missing?
Help keep Medical Image Segmentation benchmarks accurate. Report outdated results, missing benchmarks, or errors.