For missense variants on the TSC2 protein, the validation script prohibits negative predicted values. I understand that, as explained in the Wiki, "0 = unstable, 1 = wildtype, and >1 denotes predicted stability greater than wildtype". However, similar DMS experiments, such as the related challenges "PTEN and TPMT in CAGI5", display a distribution with some values below zero. Thus, a predictor targeting the same assay that in TSC2 is also expected to yield some negative values. A question thus naturally arises: when evaluating, each option seems to have a drawback, so are you going to: (1) clip negative experimental values to zero (thus losing some information in the process), (2) rescale the distribution around 1 to avoid negative values (thus leading to a different distribution from the ones in related challenges), (3) leave the experimental values as they are (thus comparing X with forced non-negative values to Y with negative values allowed), or (4) another option? A related question is: does it matter to replicate the experimental distribution, or will only variant ranking matter? Thank you and all the best, Matsvei Tsishyn

Created by Matsvei Tsishyn MatsveiTsishyn
Great, thank you very much.
Experimental scores might be negative. We are updating the description and validation script to reflect that. The experimental distribution will not be provided, so assessors will take that into account when evaluating. For reference on potential evaluation methods, see the CAGI6 HMBS evaluation (PMID: 39110250).
Hi @MatsveiTsishyn I am tagging the @CAGI7ChallengesOrganizers as they would be best suited to answer this.

Scaling conventions for TSC2 predictions page is loading…