Looking through the wiki, I had a few questions regarding the evaluation metrics and particulars of the submission contract. 1. TIFF vs. PNG: Which output specification is authoritative: float32 TIFF in the submission contract or uint8 PNG in the Quickstart? 2. Scoring: Will the official leaderboard use the four biological evaluation dimensions described in the main Evaluation page, or the MAE/RMSE/maximum-error/cosine benchmark described in the technical documentation? 3. Masking: Are evaluation metrics calculated within the target cell mask or over the full 512×512 image? 4. Test set composition: Will the hidden evaluation include approximately 50,000 or 100,000 cells, and how are proteins seen during training versus unseen proteins represented? 5. Stochastic predictions: How will multiple independently seeded predictions be requested and evaluated when the submission interface doesn't explicitly provide a seed?

Created by Tyler Thompson tythompson
Hi, I am engineer on this project building the CLI so others feel free to chime in and provide extra details: 1. uint8 PNGs are the prediction shape for input and output. We will fix the public facing docs to match that. 2. To be decided. We will update this thread when that is decided. 3. Currently target cell mask is not used in the evaluation calculations, but once again this is subject to change and we will provide updates if/when they do. 4. That is the ballpark number of cells, we will provide an update when we can tell you more about this. 5. I don't I have an answer for you yet.

Evaluation Metrics and Submission Contract page is loading…