Hello,
I am trying to download some fastq files with synapse get command
```
synapse get syn18838198
```
however during download it return randomly the following error
```
Downloading [##------------------]8.95% 840.0MB/9.2GB (4.6MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628
ValueError: Failed downloading syn18838198 to /path/to/dir/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628
caused by SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again.
```
it happens also with other files ( I tried with 30 different samples)
Am I doing something wrong or is there a workaround for this ?
Thanks
Created by gianluca mattei ginlucks @ginlucks
I do not think it is anything specific to the file being downloaded, I do not encounter any issues downloading it, e.g.
```
~$ synapse --debug get syn18838198
Downloading [####################]100.00% 9.2GB/9.2GB (151.6MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 Done...
Downloaded file: swn_patient_41_blood_1.fastq.gz
Creating /home/ubuntu/swn_patient_41_blood_1.fastq.gz
```
Are you sure the stack from the command run from the test version installed was the same? It may have looked similar but it should have had some added debugging information that may shed some light on the underlying cause. If it is different at all could you post it here?
Could you also post the output of the following commands? They may help me reproduce the environment you are running on.
```
python3 --version
synapse --version
# if you are running in a conda environment
conda env export
pip list
openssl version
# from the directory you would download into/run the synapse get command
df -k .
cat /proc/cpuinfo | grep processor | wc -l
```
nothing...I get almost the same error. "Almost" because it stucks at ~ 25%.
I also tried to download the file through Pyhton but it always stacks at ~25%.
It is strange since I can download the syn3158111 (the tutorial file) @ginlucks
Unfortunately the stack trace doesn't show as much information as I was hoping for.
If possible, could you try installing an alternate test version of the Synapse python client? This version has some enhanced retry logic and some additional logging that is being evaluated for inclusion in a future version. This test version can be installed from test.pypi.org as follows:
```
pip3 install --upgrade --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple synapseclient==2.3.1.326
```
Once installed if you could reattempt the download using the debug option and if any further stack traces occur please provide them here.
```
synapse --debug get syn18838198
```
Thank you! @jordank this is the error I get ...can't understand it very well.... maybe something related to multithread ?
```
synapse --debug get syn18838198
Downloading [##------------------]8.95% 840.0MB/9.2GB (4.6MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 2021-04-01 09:25:45,244 [client:1857 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry
f'Could not download the file: {presigned_url_provider.get_info().file_name},'
synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file
self._check_for_errors(request, completed_futures)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors
raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception
ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628
```
EDIT: It returned the same error several times for just one command (synapse --debug get syn18838198), thus I reported it just once
Here the full error message:
Full error message
```
(python35) gianlucam@neo:/data2-new/ShwannomatosisPapi/wes/fastq> synapse --debug get syn18838198
Downloading [##------------------]8.95% 840.0MB/9.2GB (4.6MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 2021-04-01 09:25:45,244 [client:1857 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry
f'Could not download the file: {presigned_url_provider.get_info().file_name},'
synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file
self._check_for_errors(request, completed_futures)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors
raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception
ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628
Downloading [##------------------]8.95% 840.0MB/9.2GB (4.6MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 2021-04-01 09:28:50,424 [client:1857 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry
f'Could not download the file: {presigned_url_provider.get_info().file_name},'
synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file
self._check_for_errors(request, completed_futures)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors
raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception
ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628
Downloading [##------------------]8.95% 840.0MB/9.2GB (4.3MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 2021-04-01 09:32:09,151 [client:1857 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry
f'Could not download the file: {presigned_url_provider.get_info().file_name},'
synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file
self._check_for_errors(request, completed_futures)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors
raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception
ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628
Downloading [##------------------]8.95% 840.0MB/9.2GB (4.2MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 2021-04-01 09:35:28,942 [client:1857 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry
f'Could not download the file: {presigned_url_provider.get_info().file_name},'
synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file
self._check_for_errors(request, completed_futures)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors
raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception
ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628
Downloading [##------------------]8.95% 840.0MB/9.2GB (4.5MB/s) swn_patient_41_blood_1.fastq.gz.synapse_download_66833628 2021-04-01 09:38:38,774 [client:1857 - DEBUG]:
Retrying download on error: [] after progressing 0 bytes
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry
f'Could not download the file: {presigned_url_provider.get_info().file_name},'
synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file
self._check_for_errors(request, completed_futures)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors
raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception
ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 334, in _get_response_with_retry
f'Could not download the file: {presigned_url_provider.get_info().file_name},'
synapseclient.core.exceptions.SynapseError: Could not download the file: swn_patient_41_blood_1.fastq.gz, please try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/user/tools/anaconda2/envs/python35/bin/synapse", line 8, in
sys.exit(main())
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/__main__.py", line 1059, in main
perform_main(args, syn)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/__main__.py", line 972, in perform_main
args.func(args, syn)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/__main__.py", line 112, in get
downloadLocation=args.downloadLocation)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 713, in get
return self._getWithEntityBundle(entityBundle=bundle, entity=entity, **kwargs)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 829, in _getWithEntityBundle
self._download_file_entity(downloadLocation, entity, ifcollision, submission)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 891, in _download_file_entity
downloadPath = self._downloadFileHandle(entity.dataFileHandleId, objectId, objectType, downloadPath)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1840, in _downloadFileHandle
expected_md5=fileHandle.get('contentMd5'))
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/client.py", line 1881, in _download_from_url_multi_threaded
multithread_download.download_file(self, request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 232, in download_file
downloader.download_file(download_request)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 309, in download_file
self._check_for_errors(request, completed_futures)
File "/home/user/tools/anaconda2/envs/python35/lib/python3.7/site-packages/synapseclient/core/multithread_download/download_threads.py", line 391, in _check_for_errors
raise ValueError(f"Failed downloading {request.object_id} to {request.path}") from exception
ValueError: Failed downloading syn18838198 to /data2-new/ShwannomatosisPapi/wes/fastq/swn_patient_41_blood_1.fastq.gz.synapse_download_66833628
```
@ginlucks If you are able to reliably reproduce this (sounds like you are) could you try this download from the command line using the --debug option and include the stack trace that is displayed (the debug option will cause the full error trace to be displayed in the console).
e.g.
```
synapse --debug get syn18838198
```
Thanks! "/path/to/dir" is just an alias. Dont ask me why but I dont feel comfortable to write the real paths of my server usually @ginlucks I notice in the error message that you have chosen to download to a location on your system called, "/path/to/dir". Is that intentional? Does that location exist?
Drop files to upload
Cant download files using synapse get page is loading…