Hi Matte, I was wondering if there is a way to download all of the BAM files at the same time given that downloading them one-by-one is really time consuming as there are hundreds of files. Best, Abolfazl

Created by Abolfazl Doostparast a.doostparast
Hi Larsson, Thanks for this brilliant tip!!
Hi Rodrigo: I see, you are right. For uploads we parallelize and have 10 concurrent threads pushing data but for download we do not. This is an open issue as we would expect a 2-5x speedup during download (depending on connection). A cheap trick I have used similar to the model you describe with the while loop is to use xargs to parallelize the download. The parallelization is -P4 which performs 4 concurrent downloads: ``` cat ~/LIST_CONTAINING_SYN_IDS_TO_DOWNLOAD | xargs -P4 -n1 -I% synapse get % ```
Hi Larsson, I believe the question was more related to the fact that the synapse utility does not allow concomitant download of multiple files.
@rodrigo.duarte and @a.doostparast If the files are accessible by a query, i.e. if you can run a query to limit the list in syn11638850 that you want you can use the command line client: ``` synapse get -q "select id from syn11638850 where <...>" ``` to do bulk downloading . For example to download every RNASeq bam file from the cmc project you would do: ``` synapse get -q "SELECT * FROM syn11638850 where fileFormat='bam' and assay='rnaSeq'" ``` If you want to bulk download everything in a folder you can also do recursive download ``` synapse get -r syn123 ``` where syn123 is a folder or project.
@rodrigo.duarte - thanks!
Thanks Rodrigo. I just obtained the IDs. Best, Abolfazl
I don't think the synapse utility has a function to download many files in parallel unfortunately... To download these files more quickly, I extracted the syn ID for each sample within the dataset (it's possible to do this here: https://www.synapse.org/#!Synapse:syn11638850/tables/ ). Then I split the ID names I wanted to download into 4-5 text files, and created 4-5 scripts containing "for" loops to download lists of individual IDs in parallel.. Example: ``` cat ~/LIST_CONTAINING_SYN_IDS_TO_DOWNLOAD | while read line do echo "Downloading $line..." synapse -u USERNAME -p PASSWORD get $line done ``` Not very fancy but it works.

Downloading all of the BAM files as a single batch page is loading…