Dear @LISA_mri_challenge ,
It is stated in the wiki that the ranking section of Task 2 that four criteria are used to determine the _best_ submission.
However, it is unclear how exactly the four criteria (DSC, HD, ASSD, RVE) are evaluated among participants.
For example, if participant A scores better on DSC and HD and participant B scores better on ASSD and RVE, which one ranks higher?
Is there some priority assigned to each metric?
Since we also have 95HD, does that count as a separate metric?
Does standard deviation (and by extension, margin of error) play a role?
Thank you in advance for your response.
Created by Mahbod Issaiy mahbodissaiy Thank you for the clarification! Dear @mahbodissaiy,
Thank you for your question.
The ranking methodology for Task 2 uses all five metrics (DSC, HD, HD95, ASSD, and RVE), which are normalized from 0 (best) to 1 (worst) and weighted equally when determining the final ranking. We updated the website to include HD95, which was accidentally omitted from the original documentation. Thank you for pointing this out.
In cases where performance across the metrics is split (e.g., one participant performs better on DSC and HD while another performs better on ASSD and RVE we will utilize the 5th metric for a tie breaker, but still treat each metric with equal importance.
If two submissions are completely equivalent across 5 metrics, we will consider standard deviation as a tie-breaker to assess the robustness and consistency of the predictions.
We hope this provides the clarity you need. Let us know if you have further questions!
Best regards,
The LISA 2025 Challenge Organizers