Dear developer,
I am sorry if the question is somewhat unclear. I am trying hard to find the best practice for the following:
##Keep my folder contents properly updated considering if I modify the folder heavily including adding a couple of files, moving around the files and folders, and deleting files.
###There are couples of upload and down options that make me sort-of confusing when to use them
- Options for upload/download
- `SyncFromSynapse`; `Synapse get -r synID (project/folder)`
- `SyncToSynapse`; `Synapse-uploader -r remote\_folder\_name synID (parent) local/path/to/folder`
- The`SyncToSynapse`/`SynFromSynapse` options seem a better way to go forward probably after the first Synapse-uploader.
- However, I felt heavy to frequently update my file structures in the manifest every time I modify. Do it mean I need to frequently create and update the manifest probably by os.walk and format the tables.
- Is there a better practice for that?
###And the problems are still lingering as the deleted files (in my local folder) will remain there in the synapse folder.
- Do I need to manually remove them or is there a better practice?
### Could you kindly inform me how Synapse recognizes the same file and control the version?
- Right now, I assumed that the Synapse recognizes the same file by file name and check the md5 to see if it is the same version?
- With the `ForceVersion`, one may control the version to change regardless of the md5 status?
- Is the mechanism the same when doing `SyncToSynapse`/`Synapse-uploader` ? I only test Synapse-uploader and it looks like it follows the first situation?
###Finally, if you can point me to useful resources, it will be great!!!
- I have checked the following but not quite got the best way to work with complex folder structures.
- **[Integrating Synapse in your RNA-Seq workflow](https://docs.synapse.org/articles/integrating_synapse_in_rna-seq_workflow.html)**: Using Synapse in an RNA-Seq workflow to manage files and track processing steps.
- **[Download Data Programmatically from Portals](https://docs.synapse.org/articles/tutorial-download-data-portal.html)**: Use R, Python or the command line to download data.
I am sorry to bombard the questions here. But I really need some input from your expertise to pull me out of the dilemma. Thanks ahead!
Best,
Minghao Gong
Created by Minghao minghao.gong@jax.org minghao.gong While we have created tools for uploading and for downloading, we have not written a tool for sync'ing. One question is whether this is truly needed: Do you need to have two copies (one on a computer file system and another in Synapse)?
> Right now, I assumed that the Synapse recognizes the same file by file name and check the md5 to see if it is the same version?
There are more details here: https://sagebionetworks.jira.com/wiki/spaces/SYNR/pages/34373660/Common+Client+Command+set+and+Cache+C4#CommonClientCommandsetandCache(%22C4%22)-CacheDesignPrinciples
> With the ForceVersion, one may control the version to change regardless of the md5 status?
Yes.
> if you can point me to useful resources, it will be great!!!
You may find these helpful:
https://python-docs.synapse.org/build/html/index.html#accessing-data
https://python-docs.synapse.org/build/html/synapseutils.html#module-synapseutils.walk
Drop files to upload
Best practice of keeping folder contents properly updated if heavily modifying the folder after a period of time. page is loading…