Hello, Thank you for running this challenge, and for the detailed answers already given on this forum - they have been genuinely useful. I have three narrow questions about the co-authorship criterion. I have read all fifteen wiki pages of the challenge project, so I am only asking about what does not appear there. **1. Is there a defined threshold for "better than random"?** The Results leaderboard carries a `P_value` column, and thread `15046` explains that the comparison to random selection is a statistical analysis on the number of identified chemical series. Is there a defined cutoff for qualifying - for example `p < 0.05` - and is it applied per submission as-is, or corrected for the number of submissions or participants? **2. Do both phases have to be cleared, or either one?** The Incentives page says *"All participants whose workflow performs better than random selection in the Blind test phase **and** Active learning phase will be invited as co-authors"*, while the main challenge page says *"...better than random in the Blind test step **or** in the Active learning step"*. Could you confirm which is intended? Specifically: if a team's blind test submission is better than random but the team does not take part in the active learning step, does that team qualify? **3. Which of a team's two test-split submissions is considered?** Teams may submit two files to the test split. Is eligibility (and the leaderboard position) based on the better of the two, or on the most recent? Thanks very much, Arnav Agarwal

Created by Arnav Agarwal ArnavAgarwal
Hello @ArnavAgarwal, 1. You can find information on how the clustering was performed [here](https://www.synapse.org/Synapse:syn75349604/discussion/threadId=15181), [here](https://www.synapse.org/Synapse:syn75349604/discussion/threadId=15113), and [here](https://www.synapse.org/Synapse:syn75349604/wiki/641046) 2. The deadline for all writeups is October 31st 3. The data folder should be accessible for all people registered to the WDR91 challenge 4. Using WDR91 data is not expected since the two proteins are unrelated. If you think there is a use for the WDR91 data in this challenge you can use them. Any external data has to be declared in the writeup.
Hello @LucaChiesa, Thank you - that settles the submission count and the revealed compounds. We will leave the revealed molecules out of our October file. A few remaining questions, whenever convenient: 1. Our earlier question on how the gold Cluster column is assigned (fingerprint, algorithm, cutoff) - is anything you can share on that? 2. Is there a deadline for the write-up of an Active Learning submission? 3. The Data folder of last year's WDR91 challenge (syn66351297) is readable but not downloadable for us. Is download access available to participants? 4. Is using the WDR91 data from AIRCHECK expected or allowed for this challenge, and if we use it, should it be declared as external_data_used in the write-up? Thank you, Arnav Agarwal
Hello @ArnavAgarwal, you can make up to two submissions in the Active Learning Step. At the moment we do not plan to release information regarding the chemical series of the hits discovered during the Blind Test Step. This might change once the full list of hits has been determined. There is no decision yet if we are going to release a subset of inactive molecules from the test set. If we were to release such molecules it will be on the same date as we release the discovered hits. The inactives are not guaranteed to be close analogs of the discovered hits, the only criteria used for their selection is popularity between the selected 50 molecules in the submissions. Revealed compounds will not be part of the test set during the Active Learning Step. The gold file containing compound labels will be updated once the molecules are released. As during the Blind Test Step (on both test split and validation split), we have included safeguards to not accept submissions containing an incorrect amount of entries or molecules not in the test set. Unless you wait until the last possible moment to make your submission, you will have plenty of time to correct you ranked candidate list.
Hello, One more, and it is a question about what you intend rather than about the data. In the active learning step, are the compounds whose activity you release on 1 October still eligible to be flagged in a team's 50 selections, or should they be excluded and the 50 drawn only from compounds that have not been tested? I ask because the evaluation function in the public Aircheck_Utils repository scores a submission by intersecting the selected set with the gold labels, and it has no notion of which compounds were previously revealed - so on the code alone a team could re-select a revealed hit and be credited with its cluster. I would rather build the file the way you mean it to be built than discover on 31 October that I had misread the step. If they are meant to be excluded I will leave them out, and it might be worth saying so on the Tasks page, since I doubt I am the only one who will wonder. Arnav Agarwal
Hello, Three short questions ahead of the active learning step, if I may. 1. How many submissions does the active learning step accept on the TEST split? The Tasks page says "You will submit 2 CSV files in the blind step and one CSV file in the active learning step", but the evaluation queue for the active learning step is configured with a total limit of 2. I would rather plan against the right number than discover it on 31 October. 2. When the hits found in participants' Blind Test selections are released on 1 October, will the release indicate which of those compounds belong to the same series or cluster? Since the criterion counts distinct clusters rather than hits, knowing whether two revealed compounds are one series or two would change how a set of 50 picks is composed. If the labels cannot be released, that is an answer too and I will infer them. 3. In thread 14646 you mentioned you might additionally release a subset of the most commonly identified false positives. Is that still planned, and would it come on the same day? Confirmed inactives near a confirmed active are as useful as the actives for judging how far a neighbourhood can be drawn, so it is worth knowing whether to expect them. Thank you again for the answers on this thread - they have settled several things for us. Arnav Agarwal
Hello, One further question about the metric, if I may. `EvaluationCode/evaluation_function.py` computes the p-value over a pre-assigned `Cluster` column of the gold set, but the repository does not include the code that produces those labels. Could you say how the clusters were assigned - which fingerprint, which algorithm, and which similarity cutoff (Butina on ECFP4 at Tanimoto 0.4, for instance)? And are the validation-split and test-split gold sets clustered the same way? The reason for asking: since the criterion depends on the number of distinct clusters hit rather than on the number of hits, knowing how a distinct series is decided would let participants reason about the composition of their 50 picks. It does not reveal which compounds are active. Thank you for the detailed answers on this thread already - they settled several questions for us. Arnav Agarwal
Hello @ArnavAgarwal, 1. After internal discussion we have settled on a p < 0.01 threshold to determine better than random submissions. The value is not corrected for the number of participants or the number of submissions. 2. Groups with better than random performance in the Blind Test Step or in the Active Learning Step will be invited as co-authors. The description in the Incentives page should be interpreted as "All participants whose workflow performs better than random selection in the Blind test phase and all participants whose workflow performs better than random selection in the Active learning phase will be invited as co-authors". 3. Both submissions are considered for eligibility. To be invited as co-authors a group needs to have at least one of its two submissions perform better than random. If both submissions performed better than random, both methods will be featured in the paper. In short, you have to identify at least two chemical series in the test split to be invited as co-author based on Blind Test Step results. Currently, five different ligands have been identified from the test split, the identity of these molecules will be revealed on October 1st (unless further review is required). All participants, regardless of previous performances, can participate. If at least one of their submissions will be better than random in the Active Learning Step, they will be invited as co-authors in the overview paper. As a reminder, submission credits on the Validation Split will be refreshed on October 1st, in other words, you do not have to save submission credits for the Active Learning Step. Unspent submission credits will not roll over to the next step.
A short follow-up on the above, in case it helps to know why the first question is the urgent one. The Results leaderboard shows 1 chemical series scoring p = 0.0491 on the test split. If the qualifying cutoff is p < 0.05 taken at face value, one series is enough; if it is corrected for the number of participants, three are needed. Those two answers imply completely different uses of a team's remaining validation credits before 30 September. Any steer, even an informal one, would be very helpful. Thank you, Arnav Agarwal

Clarification on the co-authorship criterion: threshold, and "and" vs "or" between the two phases page is loading…