Hello, I had a question regarding the evaluation and leaderboard ranking for Task 5. Several metrics are listed, including Accuracy, AUROC, F1 Score, MCC, Recall, and Specificity. Could you please clarify how the final leaderboard ranking will be determined? -Is one of these metrics the primary ranking metric? -Are the metrics listed in priority order? -Will multiple metrics be combined into a single score? -For multiclass metrics such as F1, Recall, Specificity, and AUROC, will macro, micro, or weighted averaging be used? -Since the submission format appears to contain predicted class labels rather than logits or class probabilities, how is -AUROC being calculated from these predictions? -Are one-hot encoded class predictions used for AUROC, or is there another submission format that allows probability scores or logits? Thank you.

Created by Krish Sahu lazy_lemon
Thanks for the clarification @jaydenyou!
Hi @lazy_lemon, Apologies for the confusion on the evaluation page. In our test phase, we will rank based on the aggregation of MCC and F1. It should be the mean of MCC and macro F1. The other metrics shown on the validation board provide an overall assessment of performance. With hard label prediction, the AUROC is approximated first as the mean of the sensitivity and specificity for each label, then weighted by label frequencies. The other two metrics, recall and specificity, are both class-weighted by frequencies. The F1 used in ranking and calculated in the validation board is macro F1, the unweighted mean across labels. For submission to the test phase, Docker must output a CSV file with exactly the same structure as in the validation phase. The prediction is an integer, not a probability or logits. Cheers, Jayden

[TASK 5] Metric Clarification page is loading…