Hi all, I was trying to download a large dataset with ``` synapse get -r syn51123517 ``` But the process was stuck after downloading 3TB files. And now when I try again, i am not able to initialize the downloading any more. With the command: ``` synapse --debug get -r syn51123517 ``` I got Breaking lock whose age is: 99.84932923316956 2024-01-31 11:41:01,860 [retry:125 - DEBUG]: retrying on status code: 429 2024-01-31 11:41:01,861 [retry:127 - DEBUG]: Too many concurrent requests. Allowed 3 concurrent connections at any time. 2024-01-31 11:41:01,861 [retry:165 - DEBUG]: total wait time 0 seconds ... Retrying in 0.6 seconds... Breaking lock whose age is: 99.88816475868225 2024-01-31 11:41:01,889 [retry:125 - DEBUG]: retrying on status code: 429 2024-01-31 11:41:01,889 [retry:127 - DEBUG]: Too many concurrent requests. Allowed 3 concurrent connections at any time. 2024-01-31 11:41:01,889 [retry:165 - DEBUG]: total wait time 0 seconds ... Retrying in 0.9 seconds... Breaking lock whose age is: 99.8950788974762 2024-01-31 11:41:01,897 [retry:125 - DEBUG]: retrying on status code: 429 2024-01-31 11:41:01,897 [retry:127 - DEBUG]: Too many concurrent requests. Allowed 3 concurrent connections at any time. 2024-01-31 11:41:01,898 [retry:165 - DEBUG]: total wait time 0 seconds ... Retrying in 0.6 seconds... 2024-01-31 11:41:01,901 [retry:125 - DEBUG]: retrying on status code: 429 2024-01-31 11:41:01,901 [retry:125 - DEBUG]: retrying on status code: 429 2024-01-31 11:41:01,901 [retry:127 - DEBUG]: Too many concurrent requests. Allowed 3 concurrent connections at any time. 2024-01-31 11:41:01,902 [retry:127 - DEBUG]: Too many concurrent requests. Allowed 3 concurrent connections at any time. 2024-01-31 11:41:01,902 [retry:165 - DEBUG]: total wait time 0 seconds ... Retrying in 0.8 seconds... 2024-01-31 11:41:01,902 [retry:165 - DEBUG]: total wait time 0 seconds ... Retrying in 1.0 seconds... Breaking lock whose age is: 99.92270946502686 Breaking lock whose age is: 99.92761421203613 Breaking lock whose age is: 99.94294953346252 Breaking lock whose age is: 98.95390582084656 Breaking lock whose age is: 99.96087265014648 Breaking lock whose age is: 99.97191953659058 Breaking lock whose age is: 99.9609010219574 Breaking lock whose age is: 99.98758721351624 Breaking lock whose age is: 98.98863506317139 Breaking lock whose age is: 98.99593591690063 Breaking lock whose age is: 100.85131502151489 Breaking lock whose age is: 98.718923330307 Breaking lock whose age is: 100.47487044334412 Breaking lock whose age is: 99.0100154876709 Breaking lock whose age is: 102.27163100242615 Breaking lock whose age is: 100.87758231163025 Breaking lock whose age is: 99.44224381446838 Breaking lock whose age is: 102.4751091003418 Breaking lock whose age is: 100.53278946876526 Breaking lock whose age is: 103.70307350158691 Breaking lock whose age is: 98.79800701141357 Breaking lock whose age is: 100.42701649665833 Breaking lock whose age is: 103.21961402893066 Terminated I am still able to download the single file within the dataset, but i dont wanna do that because there are so many files. Is there any other possible solution? Thank you!

Created by Xichen Wu xichwu
Thanks, @xichwu for confirming that the script did work for you. I updated my comment in the linked thread to include `ifcollision="keep.local"` - Great addition, thanks!
Hi, thanks for the script. I ran the script and it still tried to download the exisiting the files. The additional parameter `ifcollision="keep.local"` for the get function solves the problem. The script works really nice now. For the second option, I am afraid this is exactly what I tried in the past. however, it would terminate after a while with nothing occurring. This is why i ran it again with debug option, again, without quitting early. So i think there might be some problem with the code and I wish and trust your team could solve this! Big thanks Xichen
Hi @xichwu , In short - **You will need to run one of the options below**. Do not quit early even if it appears that nothing is occurring. 1) I wrote a short python script in this linked thread that you can try. The script I wrote in this thread works very similar to the `-r` flag in the CLI command you are running, however, it is slightly more efficient and less likely to run into issues: https://www.synapse.org/#!SynapseForum:threadId=10730&replyId=31279 . Make sure that you specify the `PATH_TO_DOWNLOAD_CONTENT_TO` to the directory where you have already started the download and update the `PARENT_CONTAINER_ID` to the ID of the project you are downloading. If you do not it may create a copy of the data on disk that you must manually delete. 2) Run `synapse --debug get -r syn51123517` command and do not quit early even if it appears that nothing is occurring. There are issues with how the current code is walking a project directory tree. I am meeting with the Synapse Platform team later this week to determine a long-term strategy to resolve this issue.
Hallo, thanks for you help! I ran the script with `DRY_RUN = True` and got a list of locks ``` /home/icb/xichen.wu/.synapseCache/790/97578790/.cacheMap.lock /home/icb/xichen.wu/.synapseCache/936/97578936/.cacheMap.lock /home/icb/xichen.wu/.synapseCache/88/97579088/.cacheMap.lock /home/icb/xichen.wu/.synapseCache/945/97578945/.cacheMap.lock /home/icb/xichen.wu/.synapseCache/81/97579081/.cacheMap.lock /home/icb/xichen.wu/.synapseCache/110/97579110/.cacheMap.lock /home/icb/xichen.wu/.synapseCache/947/97578947/.cacheMap.lock /home/icb/xichen.wu/.synapseCache/940/97578940/.cacheMap.lock ``` So i removed them with `DRY_RUN = False`. However, I still failed to download the files. ``` synapse --debug get -r syn51123517 ``` I got 2024-02-18 22:09:17,932 [retry:125 - DEBUG]: retrying on status code: 429 2024-02-18 22:09:17,932 [retry:127 - DEBUG]: Too many concurrent requests. Allowed 3 concurrent connections at any time. 2024-02-18 22:09:17,932 [retry:165 - DEBUG]: total wait time 0 seconds ... Retrying in 0.9 seconds... 2024-02-18 22:09:17,946 [retry:125 - DEBUG]: retrying on status code: 429 2024-02-18 22:09:17,946 [retry:127 - DEBUG]: Too many concurrent requests. Allowed 3 concurrent connections at any time. 2024-02-18 22:09:17,946 [retry:165 - DEBUG]: total wait time 0 seconds ... Retrying in 1.5 seconds... 2024-02-18 22:09:17,966 [retry:125 - DEBUG]: retrying on status code: 429 2024-02-18 22:09:17,966 [retry:127 - DEBUG]: Too many concurrent requests. Allowed 3 concurrent connections at any time. 2024-02-18 22:09:17,966 [retry:165 - DEBUG]: total wait time 0 seconds ... Retrying in 1.1 seconds... 2024-02-18 22:09:17,970 [retry:125 - DEBUG]: retrying on status code: 429 2024-02-18 22:09:17,970 [retry:127 - DEBUG]: Too many concurrent requests. Allowed 3 concurrent connections at any time. 2024-02-18 22:09:17,970 [retry:165 - DEBUG]: total wait time 0 seconds ... Retrying in 1.5 seconds... 2024-02-18 22:09:17,976 [retry:125 - DEBUG]: retrying on status code: 429 2024-02-18 22:09:17,976 [retry:127 - DEBUG]: Too many concurrent requests. Allowed 3 concurrent connections at any time. 2024-02-18 22:09:17,977 [retry:165 - DEBUG]: total wait time 0 seconds ... Retrying in 1.2 seconds... 2024-02-18 22:09:17,981 [retry:125 - DEBUG]: retrying on status code: 429 2024-02-18 22:09:17,981 [retry:127 - DEBUG]: Too many concurrent requests. Allowed 3 concurrent connections at any time. 2024-02-18 22:09:17,982 [retry:165 - DEBUG]: total wait time 0 seconds ... Retrying in 1.0 seconds... 2024-02-18 22:09:18,007 [retry:125 - DEBUG]: retrying on status code: 429 2024-02-18 22:09:18,007 [retry:127 - DEBUG]: Too many concurrent requests. Allowed 3 concurrent connections at any time. 2024-02-18 22:09:18,009 [retry:165 - DEBUG]: total wait time 0 seconds ... Retrying in 0.7 seconds... Terminated
Hello @xichwu , Thank you for providing the debug logs. I took a look at the code (https://github.com/Sage-Bionetworks/synapsePythonClient/blob/v4.0.0/synapseclient/core/lock.py#L63) that is writing these messages as well as the code that is calling this (https://github.com/search?q=repo%3ASage-Bionetworks%2FsynapsePythonClient+%22with+Lock%22&type=code). From what I can tell the code failed to clean up this custom built lock mechanism - which is why you are seeing these `Breaking lock whose age is: ....` messages. The lock directories can be freely deleted and it will not affect any files you have already downloaded, your cache, or any other files on your system. I wrote this simple Python 3 script. Please run this with `DRY_RUN = True` and it should print out all of the locks that still exist. If the script identified only those directories that look like `/home/username/.synapseCache/896/132735896/.cacheMap.lock` - Please change to `DRY_RUN = False` and execute the script again. This should clean-up these left-over locks. ``` import os DRY_RUN = True # Change the working directory to ~/.synapseCache os.chdir(os.path.expanduser("~/.synapseCache")) # Traverse the directory tree for dirpath, dirnames, filenames in os.walk("."): # Check if '.cacheMap.lock' is a directory if ".cacheMap.lock" in dirnames: if not DRY_RUN: print( f"Removing {os.path.abspath(os.path.join(dirpath, '.cacheMap.lock'))}" ) # Remove the directory os.rmdir(os.path.abspath(os.path.join(dirpath, ".cacheMap.lock"))) else: # Print the absolute path of the directory print(os.path.abspath(os.path.join(dirpath, ".cacheMap.lock"))) ``` **After these have been cleaned up please try to run `synapse get -r syn51123517` again and let me know if downloads have properly resumed.** You can freely ignore these messages: --- 2024-01-31 11:41:01,897 [retry:125 - DEBUG]: retrying on status code: 429 2024-01-31 11:41:01,897 [retry:127 - DEBUG]: Too many concurrent requests. Allowed 3 concurrent connections at any time. 2024-01-31 11:41:01,898 [retry:165 - DEBUG]: total wait time 0 seconds ---
@bfauble Can you help?

Got error Breaking lock whose age is xxx page is loading…