Hi,
I wrote a python script to download over 100 files in batch using the synapse client. However, after I successfully download several files, I got the following error message:
File "/apps/hpc/Libs/SYNAPSE/1.5.1/lib/python2.7/site-packages/synapseclient-1.5.1-py2.7.egg/synapseclient/client.py", line 691, in get
return self._getWithEntityBundle(entityBundle=bundle, entity=entity, **kwargs)
File "/apps/hpc/Libs/SYNAPSE/1.5.1/lib/python2.7/site-packages/synapseclient-1.5.1-py2.7.egg/synapseclient/client.py", line 866, in _getWithEntityBundle
entity.update(self._downloadFileEntity(entity, downloadPath, submission))
File "/apps/hpc/Libs/SYNAPSE/1.5.1/lib/python2.7/site-packages/synapseclient-1.5.1-py2.7.egg/synapseclient/client.py", line 1748, in _downloadFileEntity
return self._downloadFile(url, destination)
File "/apps/hpc/Libs/SYNAPSE/1.5.1/lib/python2.7/site-packages/synapseclient-1.5.1-py2.7.egg/synapseclient/client.py", line 1813, in _downloadFile
for nChunks, chunk in enumerate(response.iter_content(FILE_BUFFER_SIZE)):
File "/apps/hpc/Libs/SYNAPSE/1.5.1/lib/python2.7/site-packages/requests-2.10.0-py2.7.egg/requests/models.py", line 667, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: error(104, 'Connection reset by peer')", error(104, 'Connection reset by peer'))
Could anyone please help with this issue?
Thanks very much!
Created by Shuang Liu sliu Hi Sliu:
We released the client about a month ago. It can be installed with:
```
pip install -upgrade synapseclient
```
Hi Larsson,
Is the pre-release version of the client ready? Thanks very much!
I might be able to get you a pre-release version of the client. Let me get back to you. Thanks. However, right now I got this error overtime after I successfully download one file (size around 6-7G). So I have to re-run the script for every file I need to download. Moreover, I have already successfully downloaded 300+ files with each file size smaller than 2G. So this error occurred only when I tried to download for larger files. Would there be any solutions for downloading large bam files in batch before the next release of synapse client? Hi:
I know this is not much of an answer - since successful downloads are tracked you should be able to retry the command causing only the failed file to download. The issues stems upstream from Synapse in the communication between the python client and Amazon. The new version adds retries on these errors and resumes the download if Amazon gives an error message. Unfortunately I don't know what the release schedule looks like. I will get back to you as soon as it is clear. Great! When will be the next Python client release?
Thanks very much! I believe this is a known issue that should be fixed in the next Python client release (https://sagebionetworks.jira.com/browse/SYNPY-337). I'll ask colleagues if there is any way in the meantime to get this working.
Drop files to upload
error for downloading files in batch page is loading…