Hi,
I tried to do some meta-analysis on the team performances across all sub-challenges. I did it by average across two submissions, scale AUPR per sub-challenge,average per team. here is what I get
${imageLink?synapseId=syn11580619&align=None&scale=100&responsive=true}
${imageLink?synapseId=syn11580621&align=None&scale=100&responsive=true}
${imageLink?synapseId=syn11580620&align=None&scale=100&responsive=true}
Best
Vlad
```r
#scoreboard meta-analysis
##https://www.synapse.org/#!Synapse:syn11580598
library(tidyverse)
library(stringr)
name_syn=read.csv(text='name;syn
Subchallenge 1;syn11445596
Subchallenge 2.1 - Tremor;syn11559815
Subchallenge 2.2 - Dyskinesia;syn11559820
Subchallenge 2.3 - Bradykinesia;syn11559823',sep=';')
data1=name_syn %>% plyr::dlply(~syn,function(x) {syn=x$syn[1];cbind(synTableQuery(paste("SELECT * from ",syn))@values,syn=syn)})
data2=lapply(data1,function(x) {
x=x %>% mutate(submissionId=as.character(submissionId),Team=str_trim(x$Team),Team=sub('vmorozov','Vladimir Morozov',Team),Team=sub('maxwg','Max Wang',Team),Team=sub('wangsijia1990','Sijia Wang',Team)) # collapse team name by guessing
x}) %>% bind_rows()
data2=data2 %>% mutate(Rank=ifelse(is.na(Rank),Rank_Unbiased_Subset,Rank),AUPR=ifelse(is.na(AUPR),AUPR_Unbiased_Subset,AUPR))
data2=data2 %>% group_by(Team,syn) %>% summarise(Rank=mean(Rank),AUPR=mean(AUPR)) %>% group_by(syn) %>% mutate(AUPR=scale(AUPR)) #average across two submissions, scale AUPR per sub-challenge
data3=data2 %>% group_by(Team) %>% summarise(RankMean=mean(Rank),AUPRmean=mean(AUPR),participationN=n(),mPowerParticpate=any(syn=='syn11445596')) #average per team
png('AllTeams.png',width=1200,height=600)
data3 %>% ggplot(aes(RankMean,AUPRmean,col=factor(participationN),label=Team)) +geom_point()+ geom_text(check_overlap = F)
dev.off()
png('MultiCompeteTeams_RankScore.png',width=1200,height=600)
data3 %>% filter(mPowerParticpate&participationN>1) %>% ggplot(aes(RankMean,AUPRmean,col=factor(participationN),label=Team)) +geom_point()+ geom_text(check_overlap = F)+ggtitle('Teams that participated in mPower and at least one more subchallenge')
dev.off()
png('MultiCompeteTeams_mPowerVsMean.png',width=1200,height=600)
filter(data2,syn=='syn11445596') %>% transmute(Rank_mPower=Rank,Team=Team) %>% inner_join(filter(data3,mPowerParticpate&participationN>1)) %>% ggplot(aes(Rank_mPower,RankMean,col=factor(participationN),label=Team)) +geom_point()+ geom_text(check_overlap = F)+ggtitle('Teams that participated in mPower and at least one more subchallenge')
dev.off()
```
Created by Vladimir Morozov vmorozov thanks Vladimir for verbal support as always.
i hope in community phase we can figure out a more principled method to solve the problem. i am thinking what would happen if we had used sub1 model which was trained on much larger datasets to extract feature on sub2, i.e. transfer learning, which could perhaps overcomes the overfitting issues of deep learning on small dataset.
Drop files to upload
meta-analysis on the team performances across all sub-challenges page is loading…