Hi,
The final evaluation criteria do not seem to be clear to us. The rule says that final round evaluation will be based on three held-out chromosomes only, but it asks us to submit predictions across the whole genome. One can easily trick the evaluation system by ranking predictions from the three held-out chromosomes (chr1, 8, 21) on the top of the prediction list and ranking predictions from all other chromosomes below the predictions from chr1,8,21. This would significantly boost the ROC and PRC using your benchmark method and help one become a "top performing team". However, in real applications, this is obviously problematic, because one should make predictions from all chromosomes in a uniform way. I think a fair rule would be:
Either
1) ask participants to submit two sets of predictions -- (a) predictions on chr1,8,21 only, excluding all other chromosomes; (b) predictions on the whole genome -- and only use rankings in (a) for final round evaluation and then use rankings in (b) for post-competition evaluation. Here note that in (a), predictions in other chromosomes are not included in the ranking.
Or
2) ask participants to submit whole genome predictions only, but do whole genome evaluation for the final round evaluation.
Can you clarify on this?
Created by zhicheng ji zji We are doing what you propose in 1. except there is no need for 2 submissions. Just submit the predictions across the whole genome.
We are not using predictions from chromosomes (other than the test chromosomes 1,8,21) in the final round evaluation. So whatever you submit in the other chromosomes will not count for the final round evaluation. Using only the test chromosomes allows us to evaluate perdormance on chromosomes never seen in training.
We expect participants to faithfully predict on the rest of the genome as well, although this will not count to the final round. We want predictions in the whole genome in post submission evaluation to estimate how methods would do in a real life scenario of whole genome prediction in a test cell type. Of course you could submit garbage in the other chromosomes and it would not affect the final round evaluation. But we hope folks don't that.
We could have asked for 2 separate submissions. But we'd rather have one and split it up ourselves for final round and post challenge analysis.
Drop files to upload
Question on Final Submission Evaluation page is loading…