Hi @LucaChiesa, I am trying to reproduce the LightGBM/ECFP4 baseline described on the challenge wiki — "a baseline LightGBM model trained on ECFP4 using all DEL-hits and 2 times the number of negatives identified 5 hits, corresponding to 3 chemical series, in the top 50 ranked molecules in the validation split, and 1 hit in the top 50 ranked molecules in the test split." I found some difficulties due to the lack of details. Could you please clarify them? 1. "all DEL-hits" — how many compounds were labelled positive, and how were they defined? 2. Negatives — the description gives the ratio ("2 times the number of negatives") but not the population they were drawn from. Were they sampled from the compounds present in the selection file, or from the full enumerated OpenDEL library? And could you please give us some details about the negative sampling? Finally, are there plans to release the baseline code, or the labelled training set it was built from? Thank you, -Yifan

Created by YIFAN JIANG yfjiang
Hi @LucaChiesa, Thank you — this clarifies all four points. We understand that the Oct 1 release will contain the CatalogIDs and SMILES of active compounds actually recovered in participants’ selected top-50 molecules from Blind Test submissions, rather than the complete set of true hits in the test split. For our own submitted top-50 lists, compounds absent from that released active set can then be treated as inactive by exclusion; absence for compounds that we did not submit remains uninformative. We also understand that this is the “additional data” release associated with the validation-credit reset, and that it is expected on Oct 1 in Canadian time, subject to any announced delay. This is exactly what we needed to plan the Active Learning step. Thank you again for the detailed clarification.
Hello @DoingWell, > 1. Which set of compounds does "all the hits identified from the test-split" refer to? Only the hits that appeared in participants' selected 50 molecules from Blind Test submissions. The objective is to mimic a real drug discovery campaign, where the few early hits can be used to readjust the model and workflow. If the current trend observed on the test set continuous, only a handful of compounds will be revealed, the exact number will depend on the number and quality of submissions. > Will the release include negative labels for compounds that were submitted but were not hits? We will release for sure all the discovered active compounds with their CatalogID. Once the identities of these early hits are released, each group can identify by exclusion all the inactive molecules in their selected 50 molecules. Depending on the results we might additionally release a subset of some of the most commonly identified false positives. > Is the "additional data" that resets the validation submission credits the same release? Yes, the additional data is the identity of the hits discovered in the Blind Test Step. > In what form, and when exactly? We will release the CatalogIDs of all the discovered hits with their SMILES, additional information might be provided if necessary. Data should be released during the day of October 1st based on Canadian time. Possible delays in release might be experienced if we were to observe anomalous trends in the results. In case of significant delays you will be informed promptly. I hoped this clarified any doubt about the additional data for the Active Learning Step.
Hi @LucaChiesa, Thanks for the answer on the validation-split hit identities — that part is clear. I would like to ask for clarification on the other half of that answer, because it changes how we plan the Active Learning step: > All the hits identified from the test-split during the Blind Test step will be > revealed on Oct 1st for the Active Learning step. **1. Which set of compounds does "all the hits identified from the test-split" refer to?** Two readings seem possible to us: (a) **all true hits contained in the test split** — the organisers' complete list of test-split hits, independent of what participants submitted; or (b) **only the hits that appeared in participants' Blind Test submissions** — the union of the hits actually recovered by the submitted top-50 lists. The two differ by a large factor: judging from the Blind Test leaderboard, reading (b) would amount to only a handful of compounds, whereas reading (a) would be the full hit list for that split. If the precise answer is awkward to give, even the approximate number of compounds to expect would settle it for us. **2. Will the release include negative labels for compounds that were submitted but were not hits?** For a submitted 50-compound list the aggregate score tells us how many were hits but not which ones, so a list that scored 1 hit currently leaves 49 compounds whose individual outcome is unknown. Per-compound outcomes (hit / not hit) for submitted compounds would be substantially more informative for the Active Learning step than the hit identities alone. **3. Is the "additional data" that resets the validation submission credits the same release?** The Timeline page says the Active Learning step's validation credits "will be reset after the additional data is revealed", and I want to be sure that "additional data" refers to this Oct 1st test-split release rather than to a separate dataset. **4. In what form, and when exactly?** A Synapse file with CatalogIDs, or summary counts only? And will it be available at the start of the Oct 1 – Oct 31 window, so that it can inform submissions made within that window? Thank you very much.
Hello @jiaosc012, On the point 3 of your question: the identities of the validation-split hits will be revealed only after the end of the challenge (including the prospective phase). All the hits identified from the test-split during the Blind Test step will be revealed on Oct 1st for the Active Learning step.
Hi @ShayReza, @vonboss, @LucaChiesa, Thanks again for the earlier answers — the negative-ratio and the global-vs-per-sub-library points were both very helpful, and we have applied them. One question from this thread is, I think, still open, and it is the only unknown input left for us: **1. The baseline's actual positive set.** @vonboss's table was explicitly a *recommendation* ("I would recommend something like this", combining Baylor/Pfizer/your own experience), and it differs slightly from the BCM recommendation printed on the Hit Selection wiki page (`count_PGK2 >= 3`, `count_NTC = 0` there vs `> 3`, `== 0 OR < 0.1 x count_PGK2` in the post). Could you say which thresholds were actually used to build the positive set for the LightGBM/ECFP4 baseline that scored 5 hits / 3 series in the validation top-50? **If the exact values are awkward to share, just the resulting count of labelled positives would settle it for us** — a single integer. For reference, the wiki recipe gives us 26,466 compounds and the recommendation in the post gives 13,184, so the number alone would tell us which side we are on. **2. Sub-library imbalance.** Applying a global threshold to the deduplicated selection file, we find the positive rate differs by ≈185x across sub-libraries (from ≈0.002% in one to ≈0.37% in another), and two sub-libraries account for ≈70% of all positives. @vonboss noted earlier in the z-score thread that computing enrichment separately per sub-library is common practice in DEL analysis. **Was any sub-library normalisation or balancing applied when building the baseline's training set, or were the positives taken as-is after the global threshold?** **3. (Optional, low priority.)** Is there any plan to release the identities of the validation-split hits after the 15 Sep deadline? Aggregate leaderboard counts are enough for ranking, but per-compound feedback would make the active-learning step much more informative for everyone. Thanks very much,
@ShayReza @yfjiang There is a section in the documentation that explains the meaning and recommended setting of selection thresholds. Combining the recommendations from Baylor, Pfizer, and our own experience with the data, I would recommend something like this: | Criterion | Threshold | | --------------------------- | ---------------------------- | | `count_PGK2` | `> 3` | | `count_PGK2_with_inhibitor` | `== 0` OR `< 0.1 × count_PGK2` | | `count_NTC` | `== 0` OR `< 0.1 × count_PGK2` | | `historic_hits` | `< 5` |
Welcome! @vonboss Could you please provide your recommendation for the thresholds?
Thanks for the clarification. @ShayReza As @jiaosc012 mentioned, could you please provide the actual threshold values for obtaining the positive set?
For your third question, the thresholds were applied globally and were not sub-library dependent.
@jiaosc012 For your first question, we used different amounts of negative data. We built the original dataset from the library file using 10× more negatives than positives. However, while developing the models, we treated the negative-to-positive ratio as a training hyperparameter and obtained our best results when using 2× more negatives than positives. Shay
Thanks a lot, @ShayReza — that's very helpful, and the pointer to the deduplication repo is much appreciated. Three short follow-ups, mainly for @vonboss: 1. There seems to be a discrepancy I'd like to resolve before building anything on it. The Tasks/Evaluation wiki says the baseline used "all DEL-hits and 2 times the number of negatives", while the description above says negatives were selected at 10x the number of positives. Are these two different baselines, or should the 10x figure supersede the wiki text? 2. For the positive set — could you give the actual threshold values used for `n_count`, `z-score` and `count_PGK2_with_inhibitor`? Knowing the direction and rough magnitude (e.g. z-score above some cutoff, `with_inhibitor` below some fraction of `n_count`) would already be enough; I'm not asking for the processed file itself. 3. Were those thresholds applied globally across the whole selection file, or computed separately per sub-library? This is the one question from my previous post that is still open, and it matters because the sub-libraries differ substantially in sequencing depth. Thanks again,
Hi all, The reason for not sharing the processed data in this challenge is to avoid biasing participants toward our approach to the problem. Instead, we provide the full dataset so participants have more flexibility to explore different approaches. Please note that the approach described below may not necessarily be the most efficient one. For the negative set, the full library was clustered using the BitBIRCH algorithm into 603,595 clusters. Then, negatives were selected at 10× the number of positives, with 80% coming from clusters similar to the positive compounds and 20% from other clusters. (Done by @nabinbagale) The positive data were selected based on this code: https://github.com/VonBoss/DREAM_DEL_deduplication, followed by applying thresholds on n_count, z-score, and Count_PGK2_with_inhibitor. @vonboss knows the details of this part better. For the baseline models, LightGBM (LGBM) was used with some hyperparameter tuning. ECFP4 Morgan fingerprints were used with radius = 2 and 2048 bits. Shay
Hi @LucaChiesa, @ShayReza, @vonboss, Following up on Yifan's questions above — I'd like to add a few smaller ones about the same baseline, in case they're quicker to answer: 1. Was the selection file deduplicated by SMILES before labelling, or used as-is? The Deduplicate Selection page leaves this to participants, so I want to be sure I'm reading the baseline description correctly. 2. For the ECFP4 features, which radius and bit size were used (e.g. radius 2 / 2048 bits)? 3. Was the DEL-hit criterion applied globally across the whole selection file, or computed separately per sub-library? @vonboss noted in the z-score thread that "it is common practice in DEL analysis to calculate enrichment metrics separately for different sub-libraries", and the sub-libraries differ substantially in depth, so a global count threshold and a per-sub-library one give quite different positive sets. Knowing which one the baseline used would help a lot. 4. @vonboss — you mentioned on 14 Aug that you had obtained the full `count_PGK2_with_inhibitor >= 1` data (regardless of `count_PGK2`), and that the organizers were deciding whether/when to release it. Is there any update on that, and on the `count_NTC >= 1` file from Baylor? Even a "not before the 15 Sep deadline" would be useful for planning. Thanks very much, (edited only to fix formatting — the underscores in the column names had been swallowed by markdown.)
Hello @yfjiang, @ShayReza was the one who performed the baseline study. She can give you more details on the procedure. Luca

Reproducing the baseline page is loading…