Dear organizers, Thank you for organizing this challenge. I would like to point out a potential issue with the current metrics. Consider a case with one large lesion and two small lesions: Prediction A: Perfectly segments the large lesion, detects one small lesion, and has no false positives. Lesion DSC = 1.0, Lesion NSD = 1.0, Small F1 = 2/3 = 0.667. Mean of the three metrics = 8/9 = 0.889. Prediction B: Perfectly segments the large lesion, detects both small lesions, but has one additional false positive. Lesion DSC = 0.5, Lesion NSD = 0.5, Small F1 = 4/5 = 0.8. Mean of the three metrics = 0.6. Prediction B has better small-lesion detection, yet the additional false positive lowers all three ranked metrics ([official code](https://github.com/BraTS/BraTS_evaluation/blob/88e3e39cd5c4137b0831345c78d16bd393624c3a/brats_evaluation/metrics_parser.py#L174)) and reduces the mean from 0.889 to 0.6. In contrast, an additional small-lesion true positive improves only Small F1. You can even come up with more extreme examples: A: Large: 1 TP, 0 FP, 0 FN; Small: 0 TP, 10 FN, 0 FP Lesion DSC = 1.0, Lesion NSD = 1.0, Small F1 = 0. Mean of the three metrics = 2/3 = 0.667. B: Large: 1 TP, 0 FP, 0 FN; Small: 5 TP, 5 FN, 2 FP Lesion DSC = 1/3 = 0.333, Lesion NSD = 1/3 = 0.333, Small F1 = 10/17 = 0.588. Mean of the three metrics = 64/153 = 0.418. This asymmetry seems to favor conservative predictions and may discourage sensitivity to small lesions. In my view, this is a fundamental flaw in the current evaluation: if small-lesion detection is genuinely important, the scoring makes the true-positive/false-positive trade-off so unfavorable that participants are strongly incentivized to suppress small-lesion predictions rather than improve sensitivity. Is this trade-off intentional? If the goal is to balance segmentation of large lesions with detection of small lesions, wouldn't it be more consistent to classify unmatched predictions by predicted volume, so that small false positives affect the detection metric and large false positives affect the segmentation metrics? Thank you for clarifying.

Created by Luc Bouteille mts027
Dear participant, Thank you for your feedback! To ensure that participants' feedback regarding the evaluation is taken into account and to ensure that no ambiguity in interpreting the ranking strategy exists, we will soon announce a comprehensive, yet clear, description of the evaluation strategy, that answers all the questions and concerns that have been raised so far. Stay tuned!

BraTS-METS Metrics Mismatch/Quirk page is loading…