Hi organizers, I'm trying to understand how "chemical series" are identified for evaluation. Since the main statistic (before ties) is "Number of chemically distinct hit series in the 50 selected molecules" this seems really important for the results. Could you please describe how exactly you did it, and share the code? I'll describe what I've tried so far. The docs on the Evaluation page say: "Chemical series were identified using the method described in this article (Kruger et al 2020, https://pubs.acs.org/doi/abs/10.1021/acs.jcim.0c00204). A chemical series is formed by all hit molecules presenting a common substructure observed in less than 100 molecules in the full 460K ASMS screening library". I got the software that goes with that article up and running (from https://github.com/rdkit/AutomatedSeriesClassification), and I can successfully run it on small datasets. However, both the run time and memory required scale with N\^2 (N=number of molecules). For 10000 molecules, it is 1200 cpu-seconds, 2.4 GBytes of RAM, and 0.8 Gbytes of disk space. Extrapolating to 460K molecules, this would need \~6 terabytes of RAM and \~2.6 million seconds, which is impractical. From the resource requirements, I assume you didn't run this software as is on the whole 460K ASMS library, or at least not on the whole library at once. The software has an argument Classification(chembldb=...) which takes the chembl27 dataset (\~1.8M molecules); this is only used the background dataset used for the "substructure observed in less than X molecules" part of the calculation, is N\^1 and happily scales to millions of molecules. The other argument is a csv file in Classification(filename=...) which contains the molecules to classify, and that one only scales to tens of thousands in practice. The N\^2 scaling seems to be inherent because one of the things the software does is calculate an N x N pairwise distance matrix between all the input molecules. From that plus what the challenge docs say, it seems you passed in the full ASMS dataset in place of chembldb (to be able to check "observed in less than 100 molecules ..."), but only passed the **hits** as the molecules to classify into series (because classifying the full dataset runs into resource limits). So reproducing the classification requires knowing all the true hits - passing a different set of predicted hits would result in different assigned series, potentially even if the predicted hits contain all the true hits. Could you confirm my reasoning above, or share any related details from the workflow that was actually used? Practically, how would you recommend selecting 50 molecules to submit from (eg) 1000 predicted hits? Many thanks, Alex

Created by Alex Izvorski memes
Hello @memes, what you described is essentially what was done for series identification. One difference, we did not use the code in the GitHub repository, but we built our own pipeline based on the code in the supplementary information of the article. The main difference is that instead of using Arthor for substructure search we used the rdSubstructLibrary from rdkit. As you said, the full 460K ASMS screening library was used as a reference instead of ChEMBL and only the hit compounds were split into series. This was done due to size constrains, and to identify common patterns that were observed within active molecules, rather than common patterns observed within the full library. Reproducing the classification requires to know all the true hits from the full test set (test split and validation split combined), so it is not possible to reproduce the results without having access to the full labelled dataset. From a practical point of view, we do not have a specific advice on how to select the 50 molecules from your top N compounds based on diversity. Initial clustering in the workflow was performed on rdkit's ECFP4 with Tanimoto similarity using agglomerative clustering with average linkage. ECFP4 similarity could be a good starting point. An alternative would be to apply the same chemical series classification algorithm on your top N ranking molecules. Very important, we have not tested how to reproduce the series classification with alternative methods or how the series classification algorithm would behave with a different input than the original hit molecules. The proposed solutions might not give the best results. Luca

Clarification on chemical series page is loading…