**[Full text of the proposal](https://www.synapse.org/Portal/filehandle?ownerId=syn5659209&ownerType=ENTITY&xsrfToken=1EA1466FCA55F7EAE33833333900F1BC&fileName=Idea3.pdf&preview=false&wikiId=414654)** ### Anonymous Review 1 and Authors Response _**Impact:** I think this an important problem, but there have been other large scale studies to predict drug target affinities._ **Response:** This is true, and we are also aware of many computational studies for predicting drug target affinities. However, the existing prediction models are often constructed and evaluated under overly-simplified data and setups that do not really reflect the real-life problems in actual drug discovery or drug repurposing applications. Therefore, even though these theoretical prediction studies and models may have a great potential for guiding the experimental bioactivity mapping, their practical benefits for the real-world applications remain largely unknown, as most modelling works lack direct experimental evaluation of their predictions on a larger scale. This is why we and also others (see e.g. Reviewer 2 and 3 overall evaluations) think that the field is now matured enough for organizing a DREAM challenge, in which the computational teams can apply their favorite prediction models they believe are theoretically ready for drug-target affinity prediction. Then, based on a large and standardized experimental dataset collected, as well as new data generated in this Challenge, the organizers can systematically evaluate the model predictions so that the community will finally know which approaches are best for producing most accurate and robust predictions, as well as exciting biomedical discoveries, for instance, in the form of novel drug repurposing opportunities.   _**Feasibility:** It is feasible, but there are some details that are missing as discussed in the detailed review. In particular, it is important to have a strategy to determine which specific targets and drugs will be tested experimentally._ **Response:** We at FIMM are specifically interested in kinase inhibitors as kinases represent an important class of targets for anticancer treatments, as well as in many other diseases, e.g., Alzheimer’s diseases, and those are the types of drugs we are testing at FIMM using our high-throughput compound testing setup (https://vimeo.com/74608507). However, we think it will be important to carefully discuss and plan the set of drugs and targets tested together with the experimental collaborators that are supposed to generate the new experimental dataset(s), in close collaboration with us and the DREAM Challenge organizers, so that all the parties will be maximally committed to the Challenge. From the modeling and prediction point of view, kinase inhibitor target activity selectivity prediction poses a relatively challenging problem, due to the high structural and sequence similarity shared by protein kinase domains, hence leading to profound target promiscuity and polypharmacological effects. However, we are also interested in many other target classes, including GPCRs, ion channels or nuclear receptors, whichever are exciting for the data generators. As biologics today represent the major type of newly approved as well as revenue-making oncology drugs, these could provide an additional interesting drug class for the Challenge, even if target selectivity typically is much clearer with these agents.   _**Overall evaluation:** The goal of this proposal is to systematically create a comprehensive dataset of drug-target interaction affinities. To this end the investigators propose to (1) use a crowd-sourcing effort to create a single data resource that would systematically curate and store drug-target datasets using a standardized format, and, (2) to collect experimental data of drug and target interactions for about 500 drugs._ _Strengths:_ _1. The proposal is well written and is aiming to address an important problem._ _2. The created datasets would serve as a rich community resource for predicting potential targets of a drug, or predicting drugs of a potential target protein._ _3. The investigators have already developed an online web platform called Drug Target Commons which can be used for crowd-sourcing the manual curation of existing drug-target interaction datasets. The availability of this web-based resource could significantly aid in the creation of curated training and test sets._ _4. Preliminary evidence that the predictions from the KronRLS model is able to identify some valid off-targets that is supported with experimental data._ _Weaknesses:_ _1. The budget mentions number of compounds but not the number of targets. It would be helpful to know this. Also the budget does not including any aspect of the DTC. Perhaps including some incentive for contributing to the DTC could help improve the crowd-sourcing effort._ **Response:** the number and type of targets depends on the number and type of drug compounds tested (and the budget available). For instance, in case we want to focus on kinase inhibitors (e.g., those ca. 500 compounds we are currently profiling at FIMM), then the target space would cover all the human protein kinases (ca. 500 protein targets). For the other compound and target classes, these numbers differ respectively. Please see Figure 2 for some examples of target numbers and promiscuity for various target classes (e.g., GPCRs, ion channels, nuclear receptors, kinases and other enzymes). However, to systematically evaluate the computational models, it will be important to collect/generate as large and standardized target selectivity data matrix as possible, which spans over multiple compound and target families to investigate also the potential of the models to predict off-target class relationships (e.g., some kinase inhibitors may also target GPCRs, and vice versa). Such cross-class-reactivity may lead to toxic side effects, but also to novel drug repurposing opportunities. As for the DrugTargetCommons (DTC) budget, we agree that an incentive for contributing to the DTC would indeed greatly help the crowd-sourced data standardization effort. The incentive could be, for instance, authorship in the planned paper as part of the DREAM Challenge Consortium. So far, DTC has been a FIMM in-house effort (involving around 25 PhD students and postdocs), but for the Challenge, we will definitely need a larger community of domain experts to collect and standardize large enough and fully-curated experimental dataset. We have already made contacts to potential collaborators who have expressed their interest in joining such as crowd-sourcing effort, including several research institutes in Europe and in the US, as well as pharma companies that are nowadays more open for open-data research collaborations. Pharma companies are in a unique position in such an effort as they have often comprehensively profiled their own compounds for on/off-target activities, resulting in large target selectivity data matrices spanning many approved drugs currently in use.   _2. The proposed model (Kroneker regularized least square) predictive model is not described in any detail and there is no linked paper that one could read to understand exactly how the model is working. The authors also mention random forests but don't report results using these models._ **Response:** We could not unfortunately detail the models in the 5-page proposal. The Kronecker regularized least squares (KronRLS) model has been described in detail in the pilot paper we cited in the proposal (please see Suppl. Methods in AddSuppFiles-2 of Pahikkala et al. Toward more realistic drug-target interaction predictions. Brief Bioinform. 2015. doi: 10.1093/bib/bbu010). However, we note that KronRLS is just one machine learning method that we have used for such prediction problem, but the computational part of the Challenge would allow the community and teams using various models to make even better predictions. We propose that we could use KronRLS as a baseline model against which all the submissions from the DREAM Challenge participants will be benchmarked. Although we believe our preliminary results are promising, it is unlikely that the quite simple KronRLS will give the best result. Random Forests (RF) were used in the pilot study merely to show that similar trends in the results under the different, practical evaluation setups were obtained using also other models than KronRLS, to avoid reporting any model-specific artifacts (the RF results were shown in Suppl. Tables in AddSuppFiles-3 of Pahikkala et al. 2015). However, even though the two models behaved rather similarly in the practical setups S1-S4, there were many differences in the actual drug-target predictions between KronRLS and RF models, and therefore it will be interesting and important to evaluate the prediction accuracy of a wide range of computational models, both statistical and machine learning methods, in this Challenge.   _3. Oftentimes a lot of auxilliary features of the drug as well as the target is used and has been shown to be useful, including some references mentioned by the investigators. There is no mention of how these features might be used._ **Response:** This is exactly the idea of the Challenge; to make a crowd-sourced assessment of the methods and features that are most predictive of drug-target affinities. We and other have used specific structural and molecular features in the past models, and we do have further ideas that might improve the results, which can be shared and provided during the Challenge start-up phase; but most likely the community will figure out even more clever features that originate from various sources of auxiliary side information, not only those provided by us, and also ways how to use these for this prediction problem.   _4. There are many regression modeling approaches and it is not clear why the KronRLS approach should be used without showing that other regression models don't perform as well._ **Response:** Like explained above, we propose that the KronRLS model could be used as a reference baseline model against which all the submissions and models from the DREAM Challenge participants will be benchmarked. We feel our preliminary results are promising, and that it is good to have such a benchmark model available also in this Challenge, but it would be very surprising if the simple KronRLS would give the best prediction results. Like the Reviewer pointed out, there are many other modelling approaches, including more advanced regression models, which will likely lead to improved results.   _5. It is not clear how the predictions generated from the KronRLS (or other model) will be used for determining the experimental dataset creation. Currently  the experimental part of the proposal and the computational part are not well integrated._ **Response:** We apologize for not being clear enough in the proposal wording: the computational models are not meant to be used for determining the experimental dataset creation in the first place; instead, the existing experimental data will be initially collected and standardized through DTC for systematic training and initial assessment of the models submitted. The new set of experimental data will be generated at the same time to objectively test the computational models in an independent compound-target data matrix not available at the moment to avoid model over-fitting. However, if possible in this Challenge, budget or time-wise, it would be exciting to have also a follow-up, post-competition phase, where the top models from the first phase will be next applied to guiding new experimental drug-target mapping efforts, e.g., in terms of prioritizing the most potent, novel off-target interactions for further experimental evaluation. This would enable iterative, model-guided data generation process, toward really demonstrating the practical utility of the model predictions in real-life application cases, both for phenotype- and target-based drug discovery applications.

Created by Chloé-Agathe Azencott caz
### Anonymous Review 3 and Authors Response _**Impact:** The topic if this Idea is highly relevant for precision medicine. Of high interest is the concept that virtually bioassays are limited to wild-type proteins while anticancer therapies would benefit from agents specifically targeting mutated proteins. Would such data be generated and used in computational prediction approach, this would lead to a substantial scientific discovery._ _**Feasibility:** The authors described 2 approaches to get the data: (1) curating existing data that are already available through their Drug Target common platform (https://drugtargetcommons.fimm.fi/) and (2) the generation of new bioassay data (with an estimated cost of 1M USD). If option 1) can be performed with a reasonable investment (infrastructure and salaries), the second option is much more exciting but also expensive. Would this proposal be accepted, a detailed budget should be provided._ **Response:** Agreed. If large enough, new experimental data matrix of compound-target interactions could be profiled in this Challenge (provided the budget and time allows), that would be ideal, as the experimental data would then be maximally standardized, provided it comes from the same assay. However, for more comprehensive experimental validation of the model predictions, we would ideally need to cover multiple compound and target classes, which each typically come with their specific experimental assays. Moreover, there are already so much compound-target bioactivity data available, some of which we have already standardized using the DTC platform, that it would feel pity not to use these as part of the model training and initial assessment. Therefore, it seems unavoidable that some degree of bioactivity data standardization will be required in any case. Our DTC infrastructure is already up-and-running at FIMM to support such crowd-sourced data collection and standardization effort. The eventual budget of this Challenge depends on many factors, including how many and what type of compound and target classes to be profiled, what assays to use, how many concentration points per compound, and whether or not to include a post-competition phase, where the top models from the first competition phase will be applied to guiding the generation and mapping of new drug-target interactions. We think it will be important to discuss and plan each of these factors carefully with all the parties involved, the experimental collaborators that are supposed to generate the new experimental datasets (approach 2), the community and the Challenge organizers, in order to make it as exciting and feasible as possible. We also think it would be beneficial to have an incentive for contributing to the crowd-sourcing data standardization effort (approach 1), e.g., authorship in the planned paper as part of the DREAM Challenge Consortium. However, this can be decided later with the Challenge organizers.   _**Overall evaluation:** This proposal is very well written and contains enough details to assess the feasibility and impact of the Idea Challenge. Improved prediction of targets for given compounds and compounds for given targets are extremely important in drug development, as accurate predictions would enable better annotation of approved drugs and computational drug repurposing. Although a plethora of computational models have been developed to achieve these objectives, the sparsity and quality of current data is the main limiting factor. The authors clearly defined four training/validation settings and showed promising results for all of them, clearly delineating the practicality and challenges for each of them._ _My only concern is that the (inevitable) noise in the bioassay experiments is not assessed or described in the proposal._ **Response:** We thank the Reviewer for appreciating our proposal. We have already a pretty good understanding of the experimental variability and noise in such bioassays, as we are profiling at FIMM both drug sensitivities and target activities, as well as have already standardized a large collection of drug-target bioactivities from several studies, assays and bioactivity endpoints (e.g., IC50, Ki, Kd, or residual activity). In these efforts, we have indeed observed relatively large variability, part of which can be controlled for by using bioassay ontologies, like our micro bioassay ontology, which standardizes the annotation of target profiling experiments in terms of the assay type and format, endpoint type, detection technology, and other key determinants of the bioactivity readout for the crowd-sourcing effort. However, it is true that some degree of experimental variability will remain in the generated/collected datasets. This is why we proposed (see page 2, approach 1) that this 'real life' dataset from the community-effort could be initially used for training and assessment of the prediction models, as the models will need to be able to deal at least with some degree of heterogeneity in the experimental data in real application cases. The top models from this initial evaluation phase could then be applied during the post-competition phase toward guiding the generation and mapping of new drug-target interactions using single assay type, e.g., in terms of prioritizing the most potent, novel off-target interactions for further experimental evaluation.
### Anonymous Review 2 and Authors Response _** Impact:**A large-scale effort to catalog drug-target interactions and develop predictive models may impact drug development._ _**Feasibility:** The project is clear and feasible. The Drug Target Commons web site, already set up by the authors, is a nice starting point for further community work. One question that comes to mind is the possible involvement of pharmaceutical companies, who certainly also have already of lot of data produced by their own screening efforts._ **Response:** Yes, we have already made some contacts to potential collaborators who have expressed their interest in joining such open-data crowd-sourcing effort for collecting and curating drug-target bioactivities. These include several research institutes both in the US and Europe (e.g., EMBL-EBI, the host of ChEMBL database), as well as pharmaceutical companies that are nowadays more open for such research collaborations (e.g., AstraZeneca and Novartis). Pharma companies are indeed in a unique position in such an effort as they have comprehensively profiled their own compounds for on/off-target activities, resulting in large target selectivity data matrices spanning many approved drugs currently in use. We are hoping that the DREAM Challenge organizers will help us making further contacts to both academic and industrial partners, in order to make the Challenge as broad and exciting as possible.   _**Overall evaluation:** Predictive drug-target interactions has been a topic with a lot of methodological development in recent years, in particular with machine learning techniques. Most publications use the same, limited databases to train and validate their models. I have the feeling that the field is mature enough for some concerted efforts toward the generation of larger datasets of good quality, to really impact the identification of new drugs with limited side effects._ **Response:** Agreed. We thank the Reviewer for appreciating the importance of this topic and proposal.

Idea 3: Mapping compound-target interactions page is loading…