Synapse Team, Unable to download longitudinal files. For a strange reason, i am unable to view any download options. In the past, was able to download 20,000 & now do not see that option.  The programmatic option also does not work either. Am I missing something? Help appreciated. Thanks, Rahul

Created by rahul pathri rpathri
@rchai Downloaded the logs for a failed submission. A detailed log which seems like an internal error: https://docs.google.com/document/d/1Z-uS38VmZLi3AR-MdejArai7ITDzWEO-Ux2hJe6aNuQ/edit?usp=sharing This one is misleading however: https://docs.google.com/document/d/1Z9DoddzyIfEgS7aOBffO6K_VBynPcLXW4ctY_F4_iNg/edit?usp=sharing help appreciated
Hello @rchai Thanks, and submitted the same to SC2
Hi @rpathri, your latest image, `docturnalsasi/xgb_tb_model_april:latest` has been uploaded to [docker.synapse.org/syn63900418/dt_tb_model:v3](https://www.synapse.org/Synapse:syn64717075).
@rchai for your kind reference, the permission issue still exists as seen from here: https://drive.google.com/file/d/1bCO6F3Y9WxEdPAADqqUM2Ol2b9gmtLbg/view?usp=sharing
@rchai Please find the new image: https://hub.docker.com/repository/docker/docturnalsasi/xgb_tb_model_april Once you confirm, shall plan next steps around the workflow submission
@rchai We have reviewed the logs (9751367_log.txt) & found the problem to be at our end. Shall send you an updated image shortly. Thanks!
@rchai A few issues: 1) Submission was accepted at 2.34 PM CST today 2) Reported as valid at 2.35 PM CST 3) Failed at 2.37 PM CST There are non-ML based preprocessing steps that are described in the TimBre_readMe which needs to be executed. The unseen TEST set needs to be processed
@rchai Thanks for the reminder. Much appreciated & submitted successfully. Please let me know if you have any questions around the pre-processing.
Hi @rpathri , just a friendly reminder: You will need to submit your Docker image on your end by following the [Step 6. Submit your Docker image](https://www.synapse.org/Synapse:syn31472953/wiki/620031), and you should be able to receive scores via email if model runs successfully.
@rchai Great thanks! Shall look forward to the prediction results.
Hi @rpathri, Yes, I pulled the latest image you linked from Docker Hub and pushed it to your Synapse project under a different repository name. Despite the name change, the image itself remains same. Here are the commands I used for your reference: ```bash docker pull docturnalsasi/rus_tb_model_march_05:latest docker tag docturnalsasi/rus_tb_model_march_05:latest docker.synapse.org/syn63900418/dt_tb_model:v2 docker push docker.synapse.org/syn63900418/dt_tb_model:v2 ```
@rchai As confirmed yesterday at point #2, the latest image is here: https://hub.docker.com/repository/docker/docturnalsasi/rus_tb_model_march_05/general Please note the image name - "rus_tb_model_march05" for your kind reference
Hi @rpathri, Thanks for the clarification. Your [latest docker image](https://hub.docker.com/repository/docker/docturnalsasi/rus_tb_model_march_05/general) has been uploaded [here](https://www.synapse.org/Synapse:syn64717075): `docker.synapse.org/syn63900418/dt_tb_model:v2`. Please feel free to submit and let me know if you need to upload a new version of image.
@rchai Thanks for the clarification. A few points: 1) The docker image currently available in synapse repository titled - docker.synapse.org/syn63900418/dt_tb_model can be considered for the month of February OR if need be, it can be ignored 2) For the month of March, the following image needs to be pushed onto the synapse repository - https://hub.docker.com/repository/docker/docturnalsasi/rus_tb_model_march_05/general & i shall submit this to the appropriate SC. This is our priority submission Lastly, both the images have similar pre-processing steps but different algorithmic approaches. Hope this helps!
Hi @rpathri, Just to clarify, your docker image was uploaded [here](https://www.synapse.org/Synapse:syn64717075). To submit your docker image, you would need to click "**Docker Repository Tools**" --> "**Submit Docker Repository to Challenge**" and then choose the queue you would like to evaluate with, like the screen shot in the [Step 6. Submit your Docker image,](https://www.synapse.org/Synapse:syn31472953/wiki/620031). If you need to re-upload a new version of the docker image, please let me know and I can help you with that. Thank you.
@rchai As an alternative, here is the docker hub repository: https://hub.docker.com/repository/docker/docturnalsasi/rus_tb_model_march_05/general
@rchai This error was seen before & seems like it has resurfaced after a successful login. Screenshot is shared below: https://drive.google.com/file/d/10fGJcPwftzOsR629W_how8aqa_hwLVW4/view?usp=sharing
@rchai - we shall submit docker image and shall reach out should there be any questions. You can ignore files on the shared folder temporarily created for these purposes
Hi @rpathri, Thanks for your patience. The workflow is now back to receiving submissions. I just checked your submissions and it seems that you didn't submit the Docker image. Please review [Step 6. Submit your Docker image](https://www.synapse.org/Synapse:syn31472953/wiki/620031), and ensure that you submit your [Docker image](https://www.synapse.org/Synapse:syn64717075).
Thank you, Sir.
Hi @rpathri, I have been granted permission and will review your submissions. I will keep you updated. Thanks for your patience.
Hi @sieberts, I don’t have access to the submission view to check @rpathri’s submission. Could you assist with @rpathri’s request? Thanks!
@rchai: Dear Ron, is it possible for you to provide the results given a sense of urgency for us to submit phase-2 results for the following event please: indiainnovationsummit.icmr.org.in/index.php Please let me know if it is a different team that provides results for the benchmarking activity CC: @sieberts Thanks in advance!
Thanks Ron. Much appreciated. Have submitted the model for SC2. Please let me know if you have any questions around the ML pre-processing steps that are included in the readMe.txt just in case
Hi @rpathri, I'm glad you were able to push the image to Docker Hub! I have also pushed your image [docturnalsasi/dt_tb_model :latest](https://hub.docker.com/r/docturnalsasi/dt_tb_model/tags) to your project [here](https://www.synapse.org/Synapse:syn64717075). For your reference, here are the commands I used to push your image to your project: ```console docker pull docturnalsasi/dt_tb_model:latest docker tag docturnalsasi/dt_tb_model:latest docker.synapse.org/syn63900418/dt_tb_model:v1 docker push docker.synapse.org/syn63900418/dt_tb_model:v1 ``` Please feel free to submit the model to your desired evaluation queue, and let me know if you have any questions.
Ron, A few conclusive points: 1) Please use the following link to access the image: https://hub.docker.com/repository/docker/docturnalsasi/dt_tb_model/general 2) If need be, the docker file was already placed in the "Push\inference_tb" folder placed on Synapse 3) Ignore the folder "2025" uploaded yesterday & if you already were using the docker file from this one, that is fine Thanks, Rahul
Ron, Have created a new folder titled "2025" & the dockerFile is available in there along with other files https://www.synapse.org/Synapse:syn64696065/
Ron, Can this be of any use in the interim while I generate the file. This is setup for public view: https://hub.docker.com/repository/docker/docturnalsasi/rus_tb_model/tags Thanks, Rahul
@rpathri Thanks for the information! Can you please provide the [Dockerfile](https://docs.docker.com/build/concepts/dockerfile/) you used to build the image? I need to build the image on my end and help you in pushing the image.
Ron, Great to hear from you & happy new year to you too. Here is the image-link we are trying to push which does not seem to work & hence shared all the files. docker.synapse.org/syn63900418/dt_tb_model:version1 Thanks, Rahul
Hi @rpathri, Happy New Year, and apologies for the late response. I noticed that you've uploaded the files for your models [here](https://www.synapse.org/Synapse:syn64287390). Is there a **Dockerfile** available that I can use to dockerize your model?
@rchai @sieberts Happy New Year - 2025 & hope you had a great holiday season. Were you able to fix the bug. We would like to decommission the infrastructure on AWS to free up the resources as we are being billed for the storage even if the instance is shutdown. Please advise. Thanks!
@rchai - Hope you were able to access the files provided and prepare what you needed to. Please let me know if you have any questions around the TimBre_readme. @sieberts - Kindly let us know when you can share the results for a planned phase-2 publication Looking forward to hearing from Sagebionetworks team. Thanks!
@rchai Synapse Docker registry still has permissions issue when we try to push the image based on your instructions around providing userId. However, we preferred the last option of uploading all the files onto the project syn63900418 & have provided you with Admin access. From an operations standpoint, please refer to the TimBre_readMe.docx for the preprocessing, model execution and results thereon. Thanks for your help.
Hi @rpathri, We recently fixed a bug related to pushing images to our Docker registry. Could you please try pushing your image to the Synapse Docker registry again? If the issue persists, you can push the image to the GitHub Container Registry by following the [instructions](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry): - Authenticate with GitHub Container Registry using your own [GitHub Personal Access Token (PAT)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic): ``` echo "" | docker login ghcr.io -u --password-stdin ``` - Build your image locally and tag it using the format below: ``` docker tag : ghcr.io///: ``` - Push the tagged image to the GitHub Container Registry: ``` docker push ghcr.io///: ``` - Share the image in the Package Settings on GitHub Alternatively, you can upload all the necessary Docker files to your project on [syn63900418](https://www.synapse.org/Synapse:syn63900418) and grant me (@rchai) write access so I can assist further. Let me know if this works or if additional details are needed!
@rchai I am not sure we are able to push image onto the GitHub. However, when we tried to upload the files, the following error appears: https://drive.google.com/file/d/1SAA34GZuWO1WPFpzxY38LhduoqTSOGPN/view?usp=sharing Is there a better way to share the files. Upload elsewhere on Synapse platform
@rchai Please make sure you provide permissions for your gitHub account for our team to push the docker image
@rchai Ok great, we will push it to ghcr followed by the syn63900418 write access Thanks, Rahul
Hi @rpathri, Sorry for the late response. Based on the error, it seems you're having trouble connecting to the Synapse Docker registry, even though you have the necessary project permissions and credentials for pushing your image. Alternatively, could I assist you in pushing your image to your project through one of the following approaches? 1. Push your image to another registry, such as [GitHub Container Registry (ghcr)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry). - Update the visibility permissions as needed (note that all ghcr images are private by default). - My GitHub username is _rrchai_. 2. Share the necessary Docker files with me so I can build the image on my end. Afterward, please grant me (@rchai) write access to your project (syn63900418) so that I can push the image to your project. Thank you
This is a test reply. It will be deleted. This reply does not tag any users. It should be sent to all users that subscribe to this thread.
@rchai 1) Project_Sharing_Settings is in the following screenshot : https://drive.google.com/file/d/1m0X7pYk9iT0Il-1FD8OVA4ZWkPyZ8Q6Q/view?usp=sharing 2) The docker registry did not prompt for any credentials when used as follows : $ docker login docker.synapse.org (am I missing something here?) 3) However, we did use rpathri and an auth code (that I used to download longitudinal datasets) & tried to push the docker image which resulted in the access issue as well Thanks!
Thanks @rpathri. Could you please also share the current permission settings (Project Tools > Project Sharing Settings) for the project (syn63900418) you’d like to push to? I see you mentioned you created the project, but I just want to ensure that you have the appropriate permissions for it. Could you also confirm you're using the same account (@rpathri) to login to the Synapse Docker Registry? Thank you!
@rchai Yes, tried with the suggested login approach as opposed to the AuthCode please see the screenshot. https://drive.google.com/file/d/1KBeItMhO620YVyOAuidZiDBt-LXLgrRj/view?usp=sharing thanks!
Yes, tried with the suggested login approach as opposed to the AuthCode please see the screenshot. https://drive.google.com/file/d/1KBeItMhO620YVyOAuidZiDBt-LXLgrRj/view?usp=sharing thanks!
Hi @rpathri, Have you tried to login to the Synapse Docker Registry before your push the image? ``` $ docker login docker.synapse.org ```
Absolutely yes. The logging was successful via AuthToken used earlier for downloading the longitudinal datasets. The docker push command was used post this step. Please refer to the original screenshot just in case. Thanks!
@rpathri - Thank you. Just to confirm, you successfully logged in before this step?
Here is the command: $ docker push docker.synapse.org/syn63900418/dt_tb_model:version1 Here is a clearer image just in case: https://drive.google.com/file/d/1ZDZqxHa7ggsxoDATxM6N8d-Hm2PqfxCQ/view?usp=sharing Thanks!
@rpathri - It's almost impossible to read the images attached. Can you post the command you used that caused the failure? This will help us to better troubleshoot. Thanks!
@sieberts Quick update from earlier messages: 1) We have masked all of the pre-processing activity & created docker images 2) Successfully able to login with the same credentials used for downloading the longitudinal data 3) The docker push command however returns an access denied (image in the link below) 4) Project id is attached just in case we are doing something wrong (image in the link below) https://drive.google.com/drive/folders/1OxEX_IrL4U_XVUII9FQ0RQJRd7pxwSfZ?usp=sharing
@sieberts Acknowledged with thanks. Will figure out a way to mask the non-ML preprocessing code
Models must be submitted for evaluation on the Holdout data in Docker format. See [the instructions](https://www.synapse.org/Synapse:syn31472953/wiki/625268) for further details. Your Docker submission does not need to be shared publicly if your method is proprietary.
@sieberts Thanks for reiterating that. 1) We plan to submit a ML model during this month that involves non-ML based pre-processing on the holdout data set that is proprietary in nature. Since we have access to 10,008 holdout StudyIds provisioned by UCSF, we can run the predictions & submit the Python ML code, binary results & probability scores (per-individual or per-cough which are identical) as done in the past. This may include Montreal data sets, but I am not sure at this time. Please let us know if this is acceptable 2) A DL model is planned for submission during next month that does not include any proprietary pre/post processing against the holdout sets Please advise on #1.
@rpathri Please see my previous message. The holdout set is not available for download. You may submit models to be evaluated against the holdout set, however. The instructions are linked in my previous message.
@sieberts Unable to access the holdout set from the link you shared. Please see screenshot below: https://drive.google.com/file/d/1g-Ga2Kf_I4oKW4VMQF9OX3_Z6sKX558h/view?usp=drive_link
Thanks for the response around scoring set & instructions to submit the subchallenges. We shall follow them accordingly.
@rpathri 1. The holdout set is not available for download. You're welcome to submit models for evaluation against the holdout set using the [instructions here](https://www.synapse.org/Synapse:syn31472953/wiki/625268). It contains data from both the UCSF (R2D2) and University of Montreal cohorts. 2. Per [the instructions](https://www.synapse.org/Synapse:syn31472953/wiki/625268) submissions are limited to one per month per subchallenge. 3. You can submit for evaluation in either or both subchallenges at your discretion.
@sieberts , Downloaded the entire training set (Longitudinal and Solicited), it took a while given our internal cloud migration. A few follow up questions: 1) Could I use the same holdout set received from UCSF folks for generating the results & probability scores or it is better to request the data on the synapse platform to avoid variance 2) We plan to submit two sets of results (Explainable ML using ensemble and DL models). Could we do that in one go or there are any restrictions as I remember seeing limitations around the number of submissions in a given month 3) Lastly, based on our earlier learnings, we would lean towards SC2 (demographic/clinical & WAV) & totally skip the SC1 approach (minimize submissions) Please advise.
Thanks & greatly appreciated. The authentication works fine and successfully ran a test download for 1k files. Please allow me some time to confirm the complete download.
@rpathri - I think I fixed the problem. Can you give it a try now?
@Hayley.Sanchez can you verify that @rpathri has access to the CODA TB data?
Very helpful & greatly appreciated. Been able to login successfully but the code results in an error message - synapseclient.core.exceptions.SynapseHTTPError: 403 Client Error: You lack READ access to the requested entity. The tokens have all permissions explicitly granted as seen below & does not work for either solicited or longitudinal. view and download Permissions: ViewDownload Last used 3 minutes ago | Created 3 minutes ago ok this is not working for some reason Permissions: ViewDownloadModify Last used 7 minutes ago | Created 10 minutes ago
@rpathri - You'll need to set up an Auth Token for login: https://python-docs.synapse.org/tutorials/authentication/ I believe password-based authentication has been deprecated in the R and Python clients.
@sieberts Thanks for the link & while running the code, the credentials are not working & currently locked out for multiple failed attempts. I am currently using the MS-Authenticator to logon to the portal & wondering if there is a dependency.
@rpathri - I would suggest downloading the data programmatically, as download will be extremely tedious and inefficient to do through the web interface. Python code to do this is available at the bottom of [this page](syn31472953/wiki/619711). The code traverses all the nested subfolders, which contain the data and downloads it.
In continuation to my previous query: The total number of files I am able to package are - "19,996 Files eligible for packaging" from both folders longitudnal_1 and longitudnal_2. How to access the remaining files Is there a way to get to all the 700k files. Thanks, Rahul

Longitudinal data page is loading…