YOLOv11m-Seg: Real-Time Semantic Segmentation of Instruments and Anatomy in Laparoscopic Cholecystectomy

Abstract

The understanding of surgical scenes in real-time plays a crucial role in enhancing patient protection, and operational speed, and robotic surgical system independence. The deployment of robust systems faces three main obstacles: restricted training data availability and unbalanced class distribution, and time-sensitive prediction requirements. This study evaluates YOLOv8m-seg, YOLOv10m-seg, and YOLOv11m-seg models for laparoscopic cholecystectomy segmentation tasks using the CholecSeg8k dataset. The proposed system uses domain-specific augmentation techniques, including blood occlusion simulation and motion blur effects, to boost performance in actual surgical operating conditions. The experimental results demonstrate that YOLOv11m-seg achieves the best performance with 0.92 mAP@0.5, 0.82 mIoU, and 0.91 F1-score while outperforming SegMatch, and SID-RAS and other state-of-the-art models like RT-DETR and LoViT. This research confirms that YOLOv11m-seg as a dependable and efficient tool for surgical instrument and anatomical segmentation making it suitable for real-time operating room applications.