Hi @LucaChiesa, I am trying to reproduce the LightGBM/ECFP4 baseline described on the challenge wiki — "a baseline LightGBM model trained on ECFP4 using all DEL-hits and 2 times the number of negatives identified 5 hits, corresponding to 3 chemical series, in the top 50 ranked molecules in the validation split, and 1 hit in the top 50 ranked molecules in the test split." I found some difficulties due to the lack of details. Could you please clarify them? 1. "all DEL-hits" — how many compounds were labelled positive, and how were they defined? 2. Negatives — the description gives the ratio ("2 times the number of negatives") but not the population they were drawn from. Were they sampled from the compounds present in the selection file, or from the full enumerated OpenDEL library? And could you please give us some details about the negative sampling? Finally, are there plans to release the baseline code, or the labelled training set it was built from? Thank you, -Yifan

Created by YIFAN JIANG yfjiang
Hello @jiaosc012, On the point 3 of your question: the identities of the validation-split hits will be revealed only after the end of the challenge (including the prospective phase). All the hits identified from the test-split during the Blind Test step will be revealed on Oct 1st for the Active Learning step.
Hi @ShayReza, @vonboss, @LucaChiesa, Thanks again for the earlier answers — the negative-ratio and the global-vs-per-sub-library points were both very helpful, and we have applied them. One question from this thread is, I think, still open, and it is the only unknown input left for us: **1. The baseline's actual positive set.** @vonboss's table was explicitly a *recommendation* ("I would recommend something like this", combining Baylor/Pfizer/your own experience), and it differs slightly from the BCM recommendation printed on the Hit Selection wiki page (`count_PGK2 >= 3`, `count_NTC = 0` there vs `> 3`, `== 0 OR < 0.1 x count_PGK2` in the post). Could you say which thresholds were actually used to build the positive set for the LightGBM/ECFP4 baseline that scored 5 hits / 3 series in the validation top-50? **If the exact values are awkward to share, just the resulting count of labelled positives would settle it for us** — a single integer. For reference, the wiki recipe gives us 26,466 compounds and the recommendation in the post gives 13,184, so the number alone would tell us which side we are on. **2. Sub-library imbalance.** Applying a global threshold to the deduplicated selection file, we find the positive rate differs by ≈185x across sub-libraries (from ≈0.002% in one to ≈0.37% in another), and two sub-libraries account for ≈70% of all positives. @vonboss noted earlier in the z-score thread that computing enrichment separately per sub-library is common practice in DEL analysis. **Was any sub-library normalisation or balancing applied when building the baseline's training set, or were the positives taken as-is after the global threshold?** **3. (Optional, low priority.)** Is there any plan to release the identities of the validation-split hits after the 15 Sep deadline? Aggregate leaderboard counts are enough for ranking, but per-compound feedback would make the active-learning step much more informative for everyone. Thanks very much,
@ShayReza @yfjiang There is a section in the documentation that explains the meaning and recommended setting of selection thresholds. Combining the recommendations from Baylor, Pfizer, and our own experience with the data, I would recommend something like this: | Criterion | Threshold | | --------------------------- | ---------------------------- | | `count_PGK2` | `> 3` | | `count_PGK2_with_inhibitor` | `== 0` OR `< 0.1 × count_PGK2` | | `count_NTC` | `== 0` OR `< 0.1 × count_PGK2` | | `historic_hits` | `< 5` |
Welcome! @vonboss Could you please provide your recommendation for the thresholds?
Thanks for the clarification. @ShayReza As @jiaosc012 mentioned, could you please provide the actual threshold values for obtaining the positive set?
For your third question, the thresholds were applied globally and were not sub-library dependent.
@jiaosc012 For your first question, we used different amounts of negative data. We built the original dataset from the library file using 10× more negatives than positives. However, while developing the models, we treated the negative-to-positive ratio as a training hyperparameter and obtained our best results when using 2× more negatives than positives. Shay
Thanks a lot, @ShayReza — that's very helpful, and the pointer to the deduplication repo is much appreciated. Three short follow-ups, mainly for @vonboss: 1. There seems to be a discrepancy I'd like to resolve before building anything on it. The Tasks/Evaluation wiki says the baseline used "all DEL-hits and 2 times the number of negatives", while the description above says negatives were selected at 10x the number of positives. Are these two different baselines, or should the 10x figure supersede the wiki text? 2. For the positive set — could you give the actual threshold values used for `n_count`, `z-score` and `count_PGK2_with_inhibitor`? Knowing the direction and rough magnitude (e.g. z-score above some cutoff, `with_inhibitor` below some fraction of `n_count`) would already be enough; I'm not asking for the processed file itself. 3. Were those thresholds applied globally across the whole selection file, or computed separately per sub-library? This is the one question from my previous post that is still open, and it matters because the sub-libraries differ substantially in sequencing depth. Thanks again,
Hi all, The reason for not sharing the processed data in this challenge is to avoid biasing participants toward our approach to the problem. Instead, we provide the full dataset so participants have more flexibility to explore different approaches. Please note that the approach described below may not necessarily be the most efficient one. For the negative set, the full library was clustered using the BitBIRCH algorithm into 603,595 clusters. Then, negatives were selected at 10× the number of positives, with 80% coming from clusters similar to the positive compounds and 20% from other clusters. (Done by @nabinbagale) The positive data were selected based on this code: https://github.com/VonBoss/DREAM_DEL_deduplication, followed by applying thresholds on n_count, z-score, and Count_PGK2_with_inhibitor. @vonboss knows the details of this part better. For the baseline models, LightGBM (LGBM) was used with some hyperparameter tuning. ECFP4 Morgan fingerprints were used with radius = 2 and 2048 bits. Shay
Hi @LucaChiesa, @ShayReza, @vonboss, Following up on Yifan's questions above — I'd like to add a few smaller ones about the same baseline, in case they're quicker to answer: 1. Was the selection file deduplicated by SMILES before labelling, or used as-is? The Deduplicate Selection page leaves this to participants, so I want to be sure I'm reading the baseline description correctly. 2. For the ECFP4 features, which radius and bit size were used (e.g. radius 2 / 2048 bits)? 3. Was the DEL-hit criterion applied globally across the whole selection file, or computed separately per sub-library? @vonboss noted in the z-score thread that "it is common practice in DEL analysis to calculate enrichment metrics separately for different sub-libraries", and the sub-libraries differ substantially in depth, so a global count threshold and a per-sub-library one give quite different positive sets. Knowing which one the baseline used would help a lot. 4. @vonboss — you mentioned on 14 Aug that you had obtained the full `count_PGK2_with_inhibitor >= 1` data (regardless of `count_PGK2`), and that the organizers were deciding whether/when to release it. Is there any update on that, and on the `count_NTC >= 1` file from Baylor? Even a "not before the 15 Sep deadline" would be useful for planning. Thanks very much, (edited only to fix formatting — the underscores in the column names had been swallowed by markdown.)
Hello @yfjiang, @ShayReza was the one who performed the baseline study. She can give you more details on the procedure. Luca

Reproducing the baseline page is loading…