# inpainting challenge Please follow us on [Twitter](https://twitter.com/BraTS_inpaint) and take a look at the [challenge website](https://www.synapse.org/#!Synapse:syn53708249/wiki/626323). ![challenge](https://i.imgur.com/PwLaM8q.png) # Announcements **9 April**: The challenge data is released with a soft launch. Soon the website will go online. # Questions and Answers tba

Created by Florian Kofler neuronflow
Dear @vchung, thanks a lot, sorry for the confusion.
@lulululu and team, Thanks for reaching out! I'm looking at your team submissions (9746116, 9746125) and I see the following validation error: ```text Submission must be a tarball or zipped archive containing at least one NIfTI file. ``` The email you received should've contained this error as well, and not the TOIL logs (what you have attached), since this is for the workflow, not submission. I apologize for the confusion in the error messages. As for the root of the problem, I downloaded one of your submissions manually and found that the submitted prediction is not a zipfile. I checked via both the terminal and through Python: **Terminal** ```text $ unzip -l sampling_results_395k_validation_adapted_new.zip Archive: sampling_results_395k_validation_adapted_new.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. ``` **Python** ```python >>> import zipfile >>> zipfile.is_zipfile("sampling_results_395k_validation_adapted_new.zip") False ``` Hope this helps!
@vchung , could you have a look at this one, please? https://www.synapse.org/Synapse:syn53708249/discussion/threadId=10976&replyId=32588
@ShadowTwin41 The evaluation/README.md didn't explicitly specify requirements up to now due to the implicit assumption the other notebooks were run previously. This was fixed now as well as the image size description. Thanks for pointing it out!
Dear Inpainting Team,, thank you for organizing this challenge! We just saw that the inpainting evaluation queue is available now, however, we received an "invalid submission" error. As far as we understand the logs, the error is not due to our file - could you take a look at this? Please find the logs below. Thank you in advance! STDERR: 2024-07-11T07:51:22.031763627Z c25f3757b1d6 2024-07-11 07:51:22,031 MainThread INFO cwltool: Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/brats2023-main/wf-inpainting/validation-phase.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/brats2023-main/wf-inpainting/validation-phase.cwl' STDERR: 2024-07-11T07:51:27.675669816Z c25f3757b1d6 2024-07-11 07:51:27,675 MainThread WARNING cwltool: Workflow checker warning: STDERR: 2024-07-11T07:51:27.675694461Z brats2023-main/wf-inpainting/validation-phase.cwl:59:9: Source 'filepath' of type ["null", "File"] STDERR: 2024-07-11T07:51:27.675697494Z may be incompatible STDERR: 2024-07-11T07:51:27.675699669Z brats2023-main/wf-inpainting/validation-phase.cwl:93:9: with sink 'input_file' of type "File" STDERR: 2024-07-11T07:51:27.675702204Z brats2023-main/wf-inpainting/validation-phase.cwl:59:9: Source 'filepath' of type ["null", "File"] STDERR: 2024-07-11T07:51:27.675704372Z may be incompatible STDERR: 2024-07-11T07:51:27.675706402Z brats2023-main/wf-inpainting/validation-phase.cwl:171:9: with sink 'input_file' of type "File" STDERR: 2024-07-11T07:51:27.946208519Z c25f3757b1d6 2024-07-11 07:51:27,945 MainThread INFO toil: Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe. STDERR: 2024-07-11T07:51:28.448400316Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/node-f5eb706f-56d1-40b9-a49d-c106bceaf89f-1cc402dd0e11d5ae18db04a6de87223d/tmp60wd0psr/worker_log.txt STDERR: 2024-07-11T07:51:28.638969734Z c25f3757b1d6 2024-07-11 07:51:28,638 MainThread INFO toil.leader: 0 jobs are running, 0 jobs are issued and waiting to run STDERR: 2024-07-11T07:51:28.639564908Z c25f3757b1d6 2024-07-11 07:51:28,639 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_set_permissions.cwl/instance-g8a4m_4a with job batch system ID: 1 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2024-07-11T07:51:28.639644192Z c25f3757b1d6 2024-07-11 07:51:28,639 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-hgbj_5nu with job batch system ID: 2 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2024-07-11T07:51:28.639704046Z c25f3757b1d6 2024-07-11 07:51:28,639 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-x4dppq0o with job batch system ID: 3 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2024-07-11T07:51:28.639778250Z c25f3757b1d6 2024-07-11 07:51:28,639 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_get_submission.cwl/instance-907u8ilx with job batch system ID: 4 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2024-07-11T07:51:29.126668849Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/node-f5eb706f-56d1-40b9-a49d-c106bceaf89f-1cc402dd0e11d5ae18db04a6de87223d/tmpv3ydjdnw/worker_log.txt STDERR: 2024-07-11T07:51:29.126690648Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/node-f5eb706f-56d1-40b9-a49d-c106bceaf89f-1cc402dd0e11d5ae18db04a6de87223d/tmpgcv98jvp/worker_log.txt STDERR: 2024-07-11T07:51:29.137338193Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/node-f5eb706f-56d1-40b9-a49d-c106bceaf89f-1cc402dd0e11d5ae18db04a6de87223d/tmpv_p1ga47/worker_log.txt STDERR: 2024-07-11T07:51:29.137359629Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/node-f5eb706f-56d1-40b9-a49d-c106bceaf89f-1cc402dd0e11d5ae18db04a6de87223d/tmpn_ghg8eu/worker_log.txt STDERR: 2024-07-11T07:51:32.278482458Z c25f3757b1d6 2024-07-11 07:51:32,277 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_set_permissions.cwl/instance-g8a4m_4a STDERR: 2024-07-11T07:51:32.866470599Z c25f3757b1d6 2024-07-11 07:51:32,866 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-hgbj_5nu STDERR: 2024-07-11T07:52:00.435219036Z c25f3757b1d6 2024-07-11 07:52:00,434 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_get_submission.cwl/instance-907u8ilx STDERR: 2024-07-11T07:52:00.435831216Z c25f3757b1d6 2024-07-11 07:52:00,435 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/brats2023-main/wf-inpainting/steps/validate.cwl' /app/validate.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_9e5cdc0f-4e75-483e-ba69-378c5823b6da_brats2023-main_wf-inpainting_steps_validate.cwl/instance-ewrf12xl with job batch system ID: 5 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2024-07-11T07:52:00.513180264Z c25f3757b1d6 2024-07-11 07:52:00,512 Thread-48 WARNING toil.statsAndLogging: Got message from job at time 07-11-2024 07:52:00: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job files/for-job/kind-CWLWorkflow/instance-vxo0wx75/cleanup/file-x0ask5sg/stream used 109.22% (1.1 GB [1173884928B] used, 1.0 GB [1074741824B] requested) at the end of its run. STDERR: 2024-07-11T07:52:00.945180084Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/node-f5eb706f-56d1-40b9-a49d-c106bceaf89f-1cc402dd0e11d5ae18db04a6de87223d/tmpffqmg_45/worker_log.txt STDERR: 2024-07-11T07:52:13.402501562Z c25f3757b1d6 2024-07-11 07:52:13,402 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/brats2023-main/wf-inpainting/steps/validate.cwl' /app/validate.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_9e5cdc0f-4e75-483e-ba69-378c5823b6da_brats2023-main_wf-inpainting_steps_validate.cwl/instance-ewrf12xl STDERR: 2024-07-11T07:52:13.403142023Z c25f3757b1d6 2024-07-11 07:52:13,402 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_annotate_submission.cwl/instance-spss1bap with job batch system ID: 6 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2024-07-11T07:52:13.403229010Z c25f3757b1d6 2024-07-11 07:52:13,403 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_validate_email.cwl/instance-93l6a3nn with job batch system ID: 7 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2024-07-11T07:52:13.531336282Z c25f3757b1d6 2024-07-11 07:52:13,530 Thread-48 WARNING toil.statsAndLogging: Got message from job at time 07-11-2024 07:52:13: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job files/for-job/kind-CWLWorkflow/instance-vxo0wx75/cleanup/file-iy4wtbkv/stream used 109.63% (1.1 GB [1178234880B] used, 1.0 GB [1074741824B] requested) at the end of its run. STDERR: 2024-07-11T07:52:13.895502966Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/node-f5eb706f-56d1-40b9-a49d-c106bceaf89f-1cc402dd0e11d5ae18db04a6de87223d/tmpxo_wi01x/worker_log.txt STDERR: 2024-07-11T07:52:13.904633040Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/node-f5eb706f-56d1-40b9-a49d-c106bceaf89f-1cc402dd0e11d5ae18db04a6de87223d/tmpcaqsuxl2/worker_log.txt STDERR: 2024-07-11T07:52:18.041335886Z c25f3757b1d6 2024-07-11 07:52:18,041 Thread-48 WARNING toil.statsAndLogging: Got message from job at time 07-11-2024 07:52:18: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job files/for-job/kind-CWLWorkflow/instance-vxo0wx75/cleanup/file-ofmvj5ti/stream used 219.89% (2.2 GB [2363260928B] used, 1.0 GB [1074741824B] requested) at the end of its run. STDERR: 2024-07-11T07:52:18.391269476Z c25f3757b1d6 2024-07-11 07:52:18,390 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-x4dppq0o STDERR: 2024-07-11T07:52:36.962410645Z c25f3757b1d6 2024-07-11 07:52:36,962 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_annotate_submission.cwl/instance-spss1bap STDERR: 2024-07-11T07:52:37.527324634Z c25f3757b1d6 2024-07-11 07:52:37,527 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_validate_email.cwl/instance-93l6a3nn STDERR: 2024-07-11T07:52:37.527810456Z c25f3757b1d6 2024-07-11 07:52:37,527 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-bcqiktmc with job batch system ID: 8 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2024-07-11T07:52:37.570046517Z c25f3757b1d6 2024-07-11 07:52:37,569 Thread-48 WARNING toil.statsAndLogging: Got message from job at time 07-11-2024 07:52:37: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job files/for-job/kind-CWLWorkflow/instance-vxo0wx75/cleanup/file-uyhzgu55/stream used 109.22% (1.1 GB [1173884928B] used, 1.0 GB [1074741824B] requested) at the end of its run. STDERR: 2024-07-11T07:52:38.015507314Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/node-f5eb706f-56d1-40b9-a49d-c106bceaf89f-1cc402dd0e11d5ae18db04a6de87223d/tmpj295ctpu/worker_log.txt STDERR: 2024-07-11T07:52:38.227594574Z c25f3757b1d6 2024-07-11 07:52:38,227 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-bcqiktmc STDERR: 2024-07-11T07:52:38.227988619Z c25f3757b1d6 2024-07-11 07:52:38,227 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-bcqiktmc STDERR: 2024-07-11T07:52:38.228149836Z c25f3757b1d6 2024-07-11 07:52:38,228 MainThread WARNING toil.leader: Log from job kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-bcqiktmc follows: STDERR: 2024-07-11T07:52:38.228160260Z =========> STDERR: 2024-07-11T07:52:38.228163969Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG--- STDERR: 2024-07-11T07:52:38.228166987Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe. STDERR: 2024-07-11T07:52:38.228170440Z Failed to evaluate expression: STDERR: 2024-07-11T07:52:38.228173588Z Expression evaluation error: STDERR: 2024-07-11T07:52:38.228176510Z Expecting value: line 1 column 1 (char 0) STDERR: 2024-07-11T07:52:38.228179633Z script was: STDERR: 2024-07-11T07:52:38.228182476Z 01 "use strict"; STDERR: 2024-07-11T07:52:38.228185732Z 02 var inputs = { STDERR: 2024-07-11T07:52:38.228188826Z 03 "status": "INVALID", STDERR: 2024-07-11T07:52:38.228192038Z 04 "previous_annotation_finished": true, STDERR: 2024-07-11T07:52:38.228195886Z 05 "previous_email_finished": true STDERR: 2024-07-11T07:52:38.228199654Z 06 }; STDERR: 2024-07-11T07:52:38.228202611Z 07 var self = null; STDERR: 2024-07-11T07:52:38.228205767Z 08 var runtime = { STDERR: 2024-07-11T07:52:38.228208622Z 09 "cores": 1, STDERR: 2024-07-11T07:52:38.228224436Z 10 "ram": 100.0, STDERR: 2024-07-11T07:52:38.228227698Z 11 "tmpdirSize": 1024, STDERR: 2024-07-11T07:52:38.228231034Z 12 "outdirSize": 0.95367431640625, STDERR: 2024-07-11T07:52:38.228234462Z 13 "tmpdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/tmpqfktyzsq", STDERR: 2024-07-11T07:52:38.228238266Z 14 "outdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/node-f5eb706f-56d1-40b9-a49d-c106bceaf89f-1cc402dd0e11d5ae18db04a6de87223d/tmpj295ctpu/52127062-0ce5-43df-99b9-d6ef9a37ce95/tax4aw30i/tmp-out2oitipvk" STDERR: 2024-07-11T07:52:38.228242111Z 15 }; STDERR: 2024-07-11T07:52:38.228245345Z 16 (function(){ STDERR: 2024-07-11T07:52:38.228248502Z 17 if(inputs.status == "INVALID"){ STDERR: 2024-07-11T07:52:38.228251747Z 18 throw 'invalid submission'; STDERR: 2024-07-11T07:52:38.228254878Z 19 } else { STDERR: 2024-07-11T07:52:38.228257931Z 20 return {finished: true}; STDERR: 2024-07-11T07:52:38.228261036Z 21 } STDERR: 2024-07-11T07:52:38.228263950Z 22 })() STDERR: 2024-07-11T07:52:38.228266702Z stdout was: '' STDERR: 2024-07-11T07:52:38.228269645Z stderr was: 'invalid submission' STDERR: 2024-07-11T07:52:38.228272647Z STDERR: 2024-07-11T07:52:38.228275366Z WARNING:cwltool:Failed to evaluate expression: STDERR: 2024-07-11T07:52:38.228278348Z Expression evaluation error: STDERR: 2024-07-11T07:52:38.228281071Z Expecting value: line 1 column 1 (char 0) STDERR: 2024-07-11T07:52:38.228283949Z script was: STDERR: 2024-07-11T07:52:38.228287008Z 01 "use strict"; STDERR: 2024-07-11T07:52:38.228290036Z 02 var inputs = { STDERR: 2024-07-11T07:52:38.228292857Z 03 "status": "INVALID", STDERR: 2024-07-11T07:52:38.228296128Z 04 "previous_annotation_finished": true, STDERR: 2024-07-11T07:52:38.228299522Z 05 "previous_email_finished": true STDERR: 2024-07-11T07:52:38.228302858Z 06 }; STDERR: 2024-07-11T07:52:38.228306385Z 07 var self = null; STDERR: 2024-07-11T07:52:38.228309381Z 08 var runtime = { STDERR: 2024-07-11T07:52:38.228312271Z 09 "cores": 1, STDERR: 2024-07-11T07:52:38.228315382Z 10 "ram": 100.0, STDERR: 2024-07-11T07:52:38.228318360Z 11 "tmpdirSize": 1024, STDERR: 2024-07-11T07:52:38.228321759Z 12 "outdirSize": 0.95367431640625, STDERR: 2024-07-11T07:52:38.228324960Z 13 "tmpdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/tmpqfktyzsq", STDERR: 2024-07-11T07:52:38.228328940Z 14 "outdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/node-f5eb706f-56d1-40b9-a49d-c106bceaf89f-1cc402dd0e11d5ae18db04a6de87223d/tmpj295ctpu/52127062-0ce5-43df-99b9-d6ef9a37ce95/tax4aw30i/tmp-out2oitipvk" STDERR: 2024-07-11T07:52:38.228337407Z 15 }; STDERR: 2024-07-11T07:52:38.228340296Z 16 (function(){ STDERR: 2024-07-11T07:52:38.228343142Z 17 if(inputs.status == "INVALID"){ STDERR: 2024-07-11T07:52:38.228346074Z 18 throw 'invalid submission'; STDERR: 2024-07-11T07:52:38.228349121Z 19 } else { STDERR: 2024-07-11T07:52:38.228351998Z 20 return {finished: true}; STDERR: 2024-07-11T07:52:38.228355082Z 21 } STDERR: 2024-07-11T07:52:38.228357961Z 22 })() STDERR: 2024-07-11T07:52:38.228361311Z stdout was: '' STDERR: 2024-07-11T07:52:38.228364385Z stderr was: 'invalid submission' STDERR: 2024-07-11T07:52:38.228367625Z STDERR: 2024-07-11T07:52:38.228370765Z Traceback (most recent call last): STDERR: 2024-07-11T07:52:38.228373940Z File "/usr/local/lib/python3.8/site-packages/toil/worker.py", line 366, in workerScript STDERR: 2024-07-11T07:52:38.228377284Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) STDERR: 2024-07-11T07:52:38.228380431Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1392, in _runner STDERR: 2024-07-11T07:52:38.228383273Z returnValues = self._run(jobGraph, fileStore) STDERR: 2024-07-11T07:52:38.228386442Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1329, in _run STDERR: 2024-07-11T07:52:38.228389644Z return self.run(fileStore) STDERR: 2024-07-11T07:52:38.228392882Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 937, in run STDERR: 2024-07-11T07:52:38.228403548Z raise cwltool.errors.WorkflowException(status) STDERR: 2024-07-11T07:52:38.228406526Z cwltool.errors.WorkflowException: permanentFail STDERR: 2024-07-11T07:52:38.228409633Z ERROR:toil.worker:Exiting the worker because of a failed job on host c25f3757b1d6 STDERR: 2024-07-11T07:52:38.228413092Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-bcqiktmc with ID kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-bcqiktmc to 0 STDERR: 2024-07-11T07:52:38.228417722Z <========= STDERR: 2024-07-11T07:52:38.228668911Z c25f3757b1d6 2024-07-11 07:52:38,228 MainThread WARNING toil.leader: Job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-bcqiktmc with ID kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-bcqiktmc is completely failed STDERR: 2024-07-11T07:52:50.090522879Z c25f3757b1d6 2024-07-11 07:52:50,090 MainThread INFO toil.leader: Finished toil run with 8 failed jobs. STDERR: 2024-07-11T07:52:50.090578893Z c25f3757b1d6 2024-07-11 07:52:50,090 MainThread INFO toil.leader: Failed jobs at end of the run: 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-x4dppq0o 'CWLWorkflow' kind-CWLWorkflow/instance-vxo0wx75 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-hgbj_5nu 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_validate_email.cwl/instance-93l6a3nn 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/brats2023-main/wf-inpainting/steps/validate.cwl' /app/validate.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_9e5cdc0f-4e75-483e-ba69-378c5823b6da_brats2023-main_wf-inpainting_steps_validate.cwl/instance-ewrf12xl 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_annotate_submission.cwl/instance-spss1bap 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-bcqiktmc 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_get_submission.cwl/instance-907u8ilx STDERR: 2024-07-11T07:52:50.103769661Z Traceback (most recent call last): STDERR: 2024-07-11T07:52:50.103786509Z File "/usr/local/bin/toil-cwl-runner", line 8, in STDERR: 2024-07-11T07:52:50.103789913Z sys.exit(main()) STDERR: 2024-07-11T07:52:50.103791967Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 1691, in main STDERR: 2024-07-11T07:52:50.103794543Z outobj = toil.start(wf1) STDERR: 2024-07-11T07:52:50.103796713Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 829, in start STDERR: 2024-07-11T07:52:50.103798738Z return self._runMainLoop(rootJobGraph) STDERR: 2024-07-11T07:52:50.103800961Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 1115, in _runMainLoop STDERR: 2024-07-11T07:52:50.103803078Z return Leader(config=self.config, STDERR: 2024-07-11T07:52:50.103805038Z File "/usr/local/lib/python3.8/site-packages/toil/leader.py", line 269, in run STDERR: 2024-07-11T07:52:50.103807140Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore) STDERR: 2024-07-11T07:52:50.103811314Z toil.leader.FailedJobsException: The job store 'file:/var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/tmps2xs9y1g' contains 8 failed jobs: 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-x4dppq0o, 'CWLWorkflow' kind-CWLWorkflow/instance-vxo0wx75, 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-hgbj_5nu, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_validate_email.cwl/instance-93l6a3nn, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/brats2023-main/wf-inpainting/steps/validate.cwl' /app/validate.py kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_9e5cdc0f-4e75-483e-ba69-378c5823b6da_brats2023-main_wf-inpainting_steps_validate.cwl/instance-ewrf12xl, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_annotate_submission.cwl/instance-spss1bap, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-bcqiktmc, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_get_submission.cwl/instance-907u8ilx STDERR: 2024-07-11T07:52:50.103828646Z Log from job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-bcqiktmc follows: STDERR: 2024-07-11T07:52:50.103832858Z =========> STDERR: 2024-07-11T07:52:50.103834902Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG--- STDERR: 2024-07-11T07:52:50.103836967Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe. STDERR: 2024-07-11T07:52:50.103839009Z Failed to evaluate expression: STDERR: 2024-07-11T07:52:50.103840943Z Expression evaluation error: STDERR: 2024-07-11T07:52:50.103842858Z Expecting value: line 1 column 1 (char 0) STDERR: 2024-07-11T07:52:50.103844842Z script was: STDERR: 2024-07-11T07:52:50.103846710Z 01 "use strict"; STDERR: 2024-07-11T07:52:50.103848790Z 02 var inputs = { STDERR: 2024-07-11T07:52:50.103850713Z 03 "status": "INVALID", STDERR: 2024-07-11T07:52:50.103853575Z 04 "previous_annotation_finished": true, STDERR: 2024-07-11T07:52:50.103856755Z 05 "previous_email_finished": true STDERR: 2024-07-11T07:52:50.103858835Z 06 }; STDERR: 2024-07-11T07:52:50.103860649Z 07 var self = null; STDERR: 2024-07-11T07:52:50.103862540Z 08 var runtime = { STDERR: 2024-07-11T07:52:50.103864376Z 09 "cores": 1, STDERR: 2024-07-11T07:52:50.103866458Z 10 "ram": 100.0, STDERR: 2024-07-11T07:52:50.103868431Z 11 "tmpdirSize": 1024, STDERR: 2024-07-11T07:52:50.103870428Z 12 "outdirSize": 0.95367431640625, STDERR: 2024-07-11T07:52:50.103872423Z 13 "tmpdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/tmpqfktyzsq", STDERR: 2024-07-11T07:52:50.103877671Z 14 "outdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/node-f5eb706f-56d1-40b9-a49d-c106bceaf89f-1cc402dd0e11d5ae18db04a6de87223d/tmpj295ctpu/52127062-0ce5-43df-99b9-d6ef9a37ce95/tax4aw30i/tmp-out2oitipvk" STDERR: 2024-07-11T07:52:50.103880092Z 15 }; STDERR: 2024-07-11T07:52:50.103881896Z 16 (function(){ STDERR: 2024-07-11T07:52:50.103883870Z 17 if(inputs.status == "INVALID"){ STDERR: 2024-07-11T07:52:50.103886260Z 18 throw 'invalid submission'; STDERR: 2024-07-11T07:52:50.103888253Z 19 } else { STDERR: 2024-07-11T07:52:50.103890353Z 20 return {finished: true}; STDERR: 2024-07-11T07:52:50.103892293Z 21 } STDERR: 2024-07-11T07:52:50.103894095Z 22 })() STDERR: 2024-07-11T07:52:50.103895931Z stdout was: '' STDERR: 2024-07-11T07:52:50.103897973Z stderr was: 'invalid submission' STDERR: 2024-07-11T07:52:50.103901204Z STDERR: 2024-07-11T07:52:50.103904514Z WARNING:cwltool:Failed to evaluate expression: STDERR: 2024-07-11T07:52:50.103908127Z Expression evaluation error: STDERR: 2024-07-11T07:52:50.103911147Z Expecting value: line 1 column 1 (char 0) STDERR: 2024-07-11T07:52:50.103913144Z script was: STDERR: 2024-07-11T07:52:50.103914978Z 01 "use strict"; STDERR: 2024-07-11T07:52:50.103916872Z 02 var inputs = { STDERR: 2024-07-11T07:52:50.103918835Z 03 "status": "INVALID", STDERR: 2024-07-11T07:52:50.103920764Z 04 "previous_annotation_finished": true, STDERR: 2024-07-11T07:52:50.103922839Z 05 "previous_email_finished": true STDERR: 2024-07-11T07:52:50.103924904Z 06 }; STDERR: 2024-07-11T07:52:50.103926806Z 07 var self = null; STDERR: 2024-07-11T07:52:50.103928719Z 08 var runtime = { STDERR: 2024-07-11T07:52:50.103930564Z 09 "cores": 1, STDERR: 2024-07-11T07:52:50.103932455Z 10 "ram": 100.0, STDERR: 2024-07-11T07:52:50.103934334Z 11 "tmpdirSize": 1024, STDERR: 2024-07-11T07:52:50.103936307Z 12 "outdirSize": 0.95367431640625, STDERR: 2024-07-11T07:52:50.103938560Z 13 "tmpdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/tmpqfktyzsq", STDERR: 2024-07-11T07:52:50.103940779Z 14 "outdir": "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/9e5cdc0f-4e75-483e-ba69-378c5823b6da/node-f5eb706f-56d1-40b9-a49d-c106bceaf89f-1cc402dd0e11d5ae18db04a6de87223d/tmpj295ctpu/52127062-0ce5-43df-99b9-d6ef9a37ce95/tax4aw30i/tmp-out2oitipvk" STDERR: 2024-07-11T07:52:50.103943117Z 15 }; STDERR: 2024-07-11T07:52:50.103945055Z 16 (function(){ STDERR: 2024-07-11T07:52:50.103946988Z 17 if(inputs.status == "INVALID"){ STDERR: 2024-07-11T07:52:50.103948918Z 18 throw 'invalid submission'; STDERR: 2024-07-11T07:52:50.103950783Z 19 } else { STDERR: 2024-07-11T07:52:50.103955724Z 20 return {finished: true}; STDERR: 2024-07-11T07:52:50.103957860Z 21 } STDERR: 2024-07-11T07:52:50.103959789Z 22 })() STDERR: 2024-07-11T07:52:50.103961747Z stdout was: '' STDERR: 2024-07-11T07:52:50.103963683Z stderr was: 'invalid submission' STDERR: 2024-07-11T07:52:50.103965559Z STDERR: 2024-07-11T07:52:50.103967358Z Traceback (most recent call last): STDERR: 2024-07-11T07:52:50.103969256Z File "/usr/local/lib/python3.8/site-packages/toil/worker.py", line 366, in workerScript STDERR: 2024-07-11T07:52:50.103971292Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) STDERR: 2024-07-11T07:52:50.103973420Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1392, in _runner STDERR: 2024-07-11T07:52:50.103975462Z returnValues = self._run(jobGraph, fileStore) STDERR: 2024-07-11T07:52:50.103977883Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1329, in _run STDERR: 2024-07-11T07:52:50.103981055Z return self.run(fileStore) STDERR: 2024-07-11T07:52:50.103984491Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 937, in run STDERR: 2024-07-11T07:52:50.103988148Z raise cwltool.errors.WorkflowException(status) STDERR: 2024-07-11T07:52:50.103990384Z cwltool.errors.WorkflowException: permanentFail STDERR: 2024-07-11T07:52:50.103992248Z ERROR:toil.worker:Exiting the worker because of a failed job on host c25f3757b1d6 STDERR: 2024-07-11T07:52:50.103994638Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v4.0/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-bcqiktmc with ID kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v4.0_cwl_check_status.cwl/instance-bcqiktmc to 0 STDERR: 2024-07-11T07:52:50.103997186Z <=========
I didn't know that I had to install it. Where is that explained? So that I don't ask any more of these questions. Also, in the evaluation notebook, it says: "Also note that your output files have to be **full size images** with dimension 250x250x144.", but aren't the images 240x240x155? Thank you!
did you install the requirements first? In particular: `pip install inpainting`
Dear all, I was trying to use the evaluation notebook (https://github.com/BraTS-inpainting/2024_challenge/blob/main/evaluation/evaluation.ipynb) however I cannot find the following import: ``` from inpainting.challenge_metrics_2023 import generate_metrics ``` where can I find it? Thank you in advance!
@vchung do we have an ETA for the validation phase? Thank you!
Dear organisers, I'm sorry to bother you, but is there a specific date for the start of validation? Thank you in advance!
@neuronflow, Thank you very much for your answer!
Thank you for your question @ShadowTwin41 . Your model should only inpaint healthy tissue, the task of the challenge is to synthesize a healthy version of a tumor brain. Happy inpainting!
Dear organisers, First of all, I would like to thank you for preparing this challenging task! I have a question regarding the performance evaluation of the reconstructions. In your paper "https://arxiv.org/pdf/2305.08992" in Figure 3 you say "The algorithms are supposed to inpaint both the masked tumor as well as the masked healthy area.". However, the performance will only be tested in random healthy regions of the validation as testing set, as you say in section 2.4 "Since ground truth data is only available for the masked regions with healthy tissue, the evaluation will be restricted to these.". My question is, should our models also be able to inpaint tumours, or to always inpain healthy tissue? Thank you for your time!

[inpainting challenge] Announcements and Q&A page is loading…