STDERR: 2020-10-03T15:33:40.838618429Z [2020-10-03T15:33:38+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- STDERR: 2020-10-03T15:33:40.838624599Z [2020-10-03T15:33:38+0000] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host 4207e15d3a9b. STDERR: 2020-10-03T15:33:40.838630539Z [2020-10-03T15:33:38+0000] [MainThread] [W] [cwltool] [job validate_docker.cwl] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance. STDERR: 2020-10-03T15:33:40.838650199Z [2020-10-03T15:33:38+0000] [MainThread] [I] [cwltool] [job validate_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/tmxn0a464/tmp-out988d1zk7$ docker \ STDERR: 2020-10-03T15:33:40.838654549Z run \ STDERR: 2020-10-03T15:33:40.838657659Z -i \ STDERR: 2020-10-03T15:33:40.838660889Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/tmxn0a464/tmp-out988d1zk7,target=/FmlgYK \ STDERR: 2020-10-03T15:33:40.838664589Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/t7jg2rovfzzdl4fq6,target=/tmp \ STDERR: 2020-10-03T15:33:40.838668959Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/tmp_r4cpe9e.tmp,target=/var/lib/cwl/stg35ebb40b-63ff-4a5b-83d6-bf99221b8d2c/.synapseConfig,readonly \ STDERR: 2020-10-03T15:33:40.838672589Z --workdir=/FmlgYK \ STDERR: 2020-10-03T15:33:40.838675549Z --read-only=true \ STDERR: 2020-10-03T15:33:40.838678469Z --user=0:0 \ STDERR: 2020-10-03T15:33:40.838681379Z --rm \ STDERR: 2020-10-03T15:33:40.838684259Z --env=TMPDIR=/tmp \ STDERR: 2020-10-03T15:33:40.838687419Z --env=HOME=/FmlgYK \ STDERR: 2020-10-03T15:33:40.838690349Z --cidfile=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/t7jg2rovfh5ax3hw4/20201003153338-874132.cid \ STDERR: 2020-10-03T15:33:40.838694119Z sagebionetworks/synapsepythonclient:v2.1.0 \ STDERR: 2020-10-03T15:33:40.838697079Z python3 \ STDERR: 2020-10-03T15:33:40.838700069Z validate_docker.py \ STDERR: 2020-10-03T15:33:40.838702968Z -p \ STDERR: 2020-10-03T15:33:40.838705998Z docker.synapse.org/syn22920709/example1_test \ STDERR: 2020-10-03T15:33:40.838708928Z -d \ STDERR: 2020-10-03T15:33:40.838711858Z sha256:c2cb9dcba2819a1ca3e7be8da49419b1ede7c6f0357c4f04fdb3ba5c5dd78c00 \ STDERR: 2020-10-03T15:33:40.838715058Z -c \ STDERR: 2020-10-03T15:33:40.838718028Z /var/lib/cwl/stg35ebb40b-63ff-4a5b-83d6-bf99221b8d2c/.synapseConfig \ STDERR: 2020-10-03T15:33:40.838721068Z -r \ STDERR: 2020-10-03T15:33:40.838723908Z results.json STDERR: 2020-10-03T15:33:40.838726918Z Traceback (most recent call last): STDERR: 2020-10-03T15:33:40.838734528Z File "validate_docker.py", line 65, in STDERR: 2020-10-03T15:33:40.838739038Z labels = blob_resp.json()['container_config']['Labels'] STDERR: 2020-10-03T15:33:40.838742008Z KeyError: 'container_config' STDERR: 2020-10-03T15:33:40.838744898Z [2020-10-03T15:33:40+0000] [MainThread] [I] [cwltool] [job validate_docker.cwl] Max memory used: 0MiB STDERR: 2020-10-03T15:33:40.838747968Z [2020-10-03T15:33:40+0000] [MainThread] [E] [cwltool] [job validate_docker.cwl] Job error: STDERR: 2020-10-03T15:33:40.838751028Z ("Error collecting output for parameter 'results':\ncovid19-challenge-master/infrastructure/validate_docker.cwl:163:7: Did not find output file with glob pattern: '['results.json']'", {}) STDERR: 2020-10-03T15:33:40.838754668Z [2020-10-03T15:33:40+0000] [MainThread] [W] [cwltool] [job validate_docker.cwl] completed permanentFail STDERR: 2020-10-03T15:33:40.838757878Z Traceback (most recent call last): STDERR: 2020-10-03T15:33:40.838761068Z File "/usr/local/lib/python3.6/site-packages/toil/worker.py", line 368, in workerScript STDERR: 2020-10-03T15:33:40.838764398Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) STDERR: 2020-10-03T15:33:40.838767598Z File "/usr/local/lib/python3.6/site-packages/toil/job.py", line 1424, in _runner STDERR: 2020-10-03T15:33:40.838770848Z returnValues = self._run(jobGraph, fileStore) STDERR: 2020-10-03T15:33:40.838773768Z File "/usr/local/lib/python3.6/site-packages/toil/job.py", line 1361, in _run STDERR: 2020-10-03T15:33:40.838776868Z return self.run(fileStore) STDERR: 2020-10-03T15:33:40.838779748Z File "/usr/local/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 988, in run STDERR: 2020-10-03T15:33:40.838783268Z raise cwltool.errors.WorkflowException(status) STDERR: 2020-10-03T15:33:40.838786188Z cwltool.errors.WorkflowException: permanentFail STDERR: 2020-10-03T15:33:40.838789188Z [2020-10-03T15:33:40+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 4207e15d3a9b STDERR: 2020-10-03T15:33:40.838792298Z <========= STDERR: 2020-10-03T15:33:40.840704147Z [2020-10-03T15:33:40+0000] [MainThread] [W] [toil.jobGraph] Due to failure we are reducing the remaining retry count of job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r to 0 STDERR: 2020-10-03T15:33:40.846059625Z [2020-10-03T15:33:40+0000] [MainThread] [W] [toil.leader] Job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r is completely failed STDERR: 2020-10-03T15:33:42.623549255Z [2020-10-03T15:33:42+0000] [MainThread] [I] [toil.leader] Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_evaluation_id.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_evaluation_id.cwl/instance-o7v_yaac STDERR: 2020-10-03T15:33:42.785191734Z [2020-10-03T15:33:42+0000] [MainThread] [I] [toil.leader] Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_dataset.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_dataset.cwl/instance-mpftthlb STDERR: 2020-10-03T15:33:42.785862160Z [2020-10-03T15:33:42+0000] [MainThread] [I] [toil.leader] Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/dockstore-tool-synapse/v0.2/cwl/synapse-get-tool.cwl' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_dockstore-tool-synapse_v0.2_cwl_synapse-get-tool.cwl/instance-1lh4sp9g with job batch system ID: 11 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:42.786023829Z [2020-10-03T15:33:42+0000] [MainThread] [I] [toil.leader] Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_annotate_submission.cwl/instance-c2d_lvrj with job batch system ID: 12 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:43.778840217Z [2020-10-03T15:33:43+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmpzjt93w06/worker_log.txt STDERR: 2020-10-03T15:33:43.781194693Z [2020-10-03T15:33:43+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp6j1197bh/worker_log.txt STDERR: 2020-10-03T15:33:45.869209523Z [2020-10-03T15:33:45+0000] [MainThread] [I] [toil.leader] Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/dockstore-tool-synapse/v0.2/cwl/synapse-get-tool.cwl' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_dockstore-tool-synapse_v0.2_cwl_synapse-get-tool.cwl/instance-1lh4sp9g STDERR: 2020-10-03T15:33:47.503587232Z [2020-10-03T15:33:47+0000] [MainThread] [I] [toil.leader] Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_annotate_submission.cwl/instance-c2d_lvrj STDERR: 2020-10-03T15:33:57.109914740Z [2020-10-03T15:33:57+0000] [MainThread] [I] [toil.leader] Finished toil run with 7 failed jobs. STDERR: 2020-10-03T15:33:57.110286658Z [2020-10-03T15:33:57+0000] [MainThread] [I] [toil.leader] Failed jobs at end of the run: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_evaluation_id.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_evaluation_id.cwl/instance-o7v_yaac 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/dockstore-tool-synapse/v0.2/cwl/synapse-get-tool.cwl' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_dockstore-tool-synapse_v0.2_cwl_synapse-get-tool.cwl/instance-1lh4sp9g 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_get_submission.cwl/instance-uiqcdfx0 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_get_docker_config.cwl/instance-wwx9ulj7 'CWLWorkflow' kind-CWLWorkflow/instance-pv636rz9 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_dataset.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_dataset.cwl/instance-mpftthlb STDERR: 2020-10-03T15:33:57.122875096Z Traceback (most recent call last): STDERR: 2020-10-03T15:33:57.122888976Z File "/usr/local/bin/toil-cwl-runner", line 8, in STDERR: 2020-10-03T15:33:57.122893376Z sys.exit(main()) STDERR: 2020-10-03T15:33:57.122896956Z File "/usr/local/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 1788, in main STDERR: 2020-10-03T15:33:57.122900576Z outobj = toil.start(wf1) STDERR: 2020-10-03T15:33:57.122903696Z File "/usr/local/lib/python3.6/site-packages/toil/common.py", line 846, in start STDERR: 2020-10-03T15:33:57.122907066Z return self._runMainLoop(rootJobGraph) STDERR: 2020-10-03T15:33:57.122910336Z File "/usr/local/lib/python3.6/site-packages/toil/common.py", line 1138, in _runMainLoop STDERR: 2020-10-03T15:33:57.122913526Z jobCache=self._jobCache).run() STDERR: 2020-10-03T15:33:57.122916386Z File "/usr/local/lib/python3.6/site-packages/toil/leader.py", line 269, in run STDERR: 2020-10-03T15:33:57.122919476Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore) STDERR: 2020-10-03T15:33:57.122923586Z toil.leader.FailedJobsException: The job store 'file:/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/tmp4n6tnd_5' contains 7 failed jobs: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_evaluation_id.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_evaluation_id.cwl/instance-o7v_yaac, 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/dockstore-tool-synapse/v0.2/cwl/synapse-get-tool.cwl' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_dockstore-tool-synapse_v0.2_cwl_synapse-get-tool.cwl/instance-1lh4sp9g, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_get_submission.cwl/instance-uiqcdfx0, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_get_docker_config.cwl/instance-wwx9ulj7, 'CWLWorkflow' kind-CWLWorkflow/instance-pv636rz9, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_dataset.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_dataset.cwl/instance-mpftthlb STDERR: 2020-10-03T15:33:57.122937876Z Log from job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r follows: STDERR: 2020-10-03T15:33:57.122941746Z =========> STDERR: 2020-10-03T15:33:57.122944736Z [2020-10-03T15:33:38+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- STDERR: 2020-10-03T15:33:57.122948256Z [2020-10-03T15:33:38+0000] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host 4207e15d3a9b. STDERR: 2020-10-03T15:33:57.122951616Z [2020-10-03T15:33:38+0000] [MainThread] [W] [cwltool] [job validate_docker.cwl] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance. STDERR: 2020-10-03T15:33:57.122955116Z [2020-10-03T15:33:38+0000] [MainThread] [I] [cwltool] [job validate_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/tmxn0a464/tmp-out988d1zk7$ docker \ STDERR: 2020-10-03T15:33:57.122958796Z run \ STDERR: 2020-10-03T15:33:57.122961846Z -i \ STDERR: 2020-10-03T15:33:57.122965486Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/tmxn0a464/tmp-out988d1zk7,target=/FmlgYK \ STDERR: 2020-10-03T15:33:57.122969236Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/t7jg2rovfzzdl4fq6,target=/tmp \ STDERR: 2020-10-03T15:33:57.122972786Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/tmp_r4cpe9e.tmp,target=/var/lib/cwl/stg35ebb40b-63ff-4a5b-83d6-bf99221b8d2c/.synapseConfig,readonly \ STDERR: 2020-10-03T15:33:57.122979686Z --workdir=/FmlgYK \ STDERR: 2020-10-03T15:33:57.122982896Z --read-only=true \ STDERR: 2020-10-03T15:33:57.122985826Z --user=0:0 \ STDERR: 2020-10-03T15:33:57.122988766Z --rm \ STDERR: 2020-10-03T15:33:57.122991656Z --env=TMPDIR=/tmp \ STDERR: 2020-10-03T15:33:57.122994626Z --env=HOME=/FmlgYK \ STDERR: 2020-10-03T15:33:57.122997536Z --cidfile=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/t7jg2rovfh5ax3hw4/20201003153338-874132.cid \ STDERR: 2020-10-03T15:33:57.123000946Z sagebionetworks/synapsepythonclient:v2.1.0 \ STDERR: 2020-10-03T15:33:57.123003916Z python3 \ STDERR: 2020-10-03T15:33:57.123006756Z validate_docker.py \ STDERR: 2020-10-03T15:33:57.123009636Z -p \ STDERR: 2020-10-03T15:33:57.123012506Z docker.synapse.org/syn22920709/example1_test \ STDERR: 2020-10-03T15:33:57.123015725Z -d \ STDERR: 2020-10-03T15:33:57.123018845Z sha256:c2cb9dcba2819a1ca3e7be8da49419b1ede7c6f0357c4f04fdb3ba5c5dd78c00 \ STDERR: 2020-10-03T15:33:57.123022125Z -c \ STDERR: 2020-10-03T15:33:57.123025165Z /var/lib/cwl/stg35ebb40b-63ff-4a5b-83d6-bf99221b8d2c/.synapseConfig \ STDERR: 2020-10-03T15:33:57.123028285Z -r \ STDERR: 2020-10-03T15:33:57.123031115Z results.json STDERR: 2020-10-03T15:33:57.123034045Z Traceback (most recent call last): STDERR: 2020-10-03T15:33:57.123037275Z File "validate_docker.py", line 65, in STDERR: 2020-10-03T15:33:57.123040595Z labels = blob_resp.json()['container_config']['Labels'] STDERR: 2020-10-03T15:33:57.123043545Z KeyError: 'container_config' STDERR: 2020-10-03T15:33:57.123046445Z [2020-10-03T15:33:40+0000] [MainThread] [I] [cwltool] [job validate_docker.cwl] Max memory used: 0MiB STDERR: 2020-10-03T15:33:57.123049515Z [2020-10-03T15:33:40+0000] [MainThread] [E] [cwltool] [job validate_docker.cwl] Job error: STDERR: 2020-10-03T15:33:57.123053145Z ("Error collecting output for parameter 'results':\ncovid19-challenge-master/infrastructure/validate_docker.cwl:163:7: Did not find output file with glob pattern: '['results.json']'", {}) STDERR: 2020-10-03T15:33:57.123056665Z [2020-10-03T15:33:40+0000] [MainThread] [W] [cwltool] [job validate_docker.cwl] completed permanentFail STDERR: 2020-10-03T15:33:57.123059725Z Traceback (most recent call last): STDERR: 2020-10-03T15:33:57.123062655Z File "/usr/local/lib/python3.6/site-packages/toil/worker.py", line 368, in workerScript STDERR: 2020-10-03T15:33:57.123065805Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) STDERR: 2020-10-03T15:33:57.123068975Z File "/usr/local/lib/python3.6/site-packages/toil/job.py", line 1424, in _runner STDERR: 2020-10-03T15:33:57.123076175Z returnValues = self._run(jobGraph, fileStore) STDERR: 2020-10-03T15:33:57.123079605Z File "/usr/local/lib/python3.6/site-packages/toil/job.py", line 1361, in _run STDERR: 2020-10-03T15:33:57.123082885Z return self.run(fileStore) STDERR: 2020-10-03T15:33:57.123085755Z File "/usr/local/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 988, in run STDERR: 2020-10-03T15:33:57.123088855Z raise cwltool.errors.WorkflowException(status) STDERR: 2020-10-03T15:33:57.123091745Z cwltool.errors.WorkflowException: permanentFail STDERR: 2020-10-03T15:33:57.123094665Z [2020-10-03T15:33:40+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 4207e15d3a9b STDERR: 2020-10-03T15:33:57.123097685Z <========= ``` " />

My model submission is producing unusual errors in the logs, so I tried submitting the example code as a test. That doesn't seem to be working either. Here is the error log. Can anyone tell what is wrong? Is there an issue for the administrators to fix? ``` STDERR: 2020-10-03T15:33:19.374332685Z [2020-10-03T15:33:19+0000] [MainThread] [I] [cwltool] Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/main_workflow.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/main_workflow.cwl' STDERR: 2020-10-03T15:33:29.577441752Z [2020-10-03T15:33:29+0000] [MainThread] [W] [cwltool] Workflow checker warning: STDERR: 2020-10-03T15:33:29.577468442Z covid19-challenge-master/infrastructure/main_workflow.cwl:90:9: Source 'docker_repository' of type STDERR: 2020-10-03T15:33:29.577473062Z ["null", "string"] may be STDERR: 2020-10-03T15:33:29.577477442Z incompatible STDERR: 2020-10-03T15:33:29.577481312Z covid19-challenge-master/infrastructure/main_workflow.cwl:242:9: with sink 'docker_repository' STDERR: 2020-10-03T15:33:29.577484972Z of type "string" STDERR: 2020-10-03T15:33:29.577488702Z covid19-challenge-master/infrastructure/main_workflow.cwl:91:9: Source 'docker_digest' of type STDERR: 2020-10-03T15:33:29.577492222Z ["null", "string"] may be STDERR: 2020-10-03T15:33:29.577495952Z incompatible STDERR: 2020-10-03T15:33:29.577499312Z covid19-challenge-master/infrastructure/main_workflow.cwl:244:9: with sink 'docker_digest' of STDERR: 2020-10-03T15:33:29.577502752Z type "string" STDERR: 2020-10-03T15:33:29.577506302Z covid19-challenge-master/infrastructure/main_workflow.cwl:252:9: 'status' is not an input parameter STDERR: 2020-10-03T15:33:29.577509802Z of STDERR: 2020-10-03T15:33:29.577513342Z file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/run_synthetic_infer_docker.cwl, STDERR: 2020-10-03T15:33:29.577517332Z expected docker_repository, STDERR: 2020-10-03T15:33:29.577521222Z docker_digest, submissionid, STDERR: 2020-10-03T15:33:29.577524942Z docker_registry, STDERR: 2020-10-03T15:33:29.577528372Z docker_authentication, parentid, STDERR: 2020-10-03T15:33:29.577531832Z synapse_config, model, input_dir, STDERR: 2020-10-03T15:33:29.577535282Z docker_script, quota STDERR: 2020-10-03T15:33:29.577538752Z covid19-challenge-master/infrastructure/main_workflow.cwl:90:9: Source 'docker_repository' of type STDERR: 2020-10-03T15:33:29.577543212Z ["null", "string"] may be STDERR: 2020-10-03T15:33:29.577559002Z incompatible STDERR: 2020-10-03T15:33:29.577562332Z covid19-challenge-master/infrastructure/main_workflow.cwl:209:9: with sink 'docker_repository' STDERR: 2020-10-03T15:33:29.577565352Z of type "string" STDERR: 2020-10-03T15:33:29.577568492Z covid19-challenge-master/infrastructure/main_workflow.cwl:91:9: Source 'docker_digest' of type STDERR: 2020-10-03T15:33:29.577571532Z ["null", "string"] may be STDERR: 2020-10-03T15:33:29.577574732Z incompatible STDERR: 2020-10-03T15:33:29.577577852Z covid19-challenge-master/infrastructure/main_workflow.cwl:211:9: with sink 'docker_digest' of STDERR: 2020-10-03T15:33:29.577581162Z type "string" STDERR: 2020-10-03T15:33:30.968922990Z [2020-10-03T15:33:30+0000] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host 4207e15d3a9b. STDERR: 2020-10-03T15:33:32.040146020Z [2020-10-03T15:33:32+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp2wqbdfzg/worker_log.txt STDERR: 2020-10-03T15:33:32.498853740Z [2020-10-03T15:33:32+0000] [MainThread] [I] [toil.leader] 0 jobs are running, 0 jobs are issued and waiting to run STDERR: 2020-10-03T15:33:32.500969452Z [2020-10-03T15:33:32+0000] [MainThread] [I] [toil.leader] Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_set_permissions.cwl/instance-g547d1ny with job batch system ID: 1 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:32.501140013Z [2020-10-03T15:33:32+0000] [MainThread] [I] [toil.leader] Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/create_adminsynid_annotation.cwl' echo kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_create_adminsynid_annotation.cwl/instance-pkfi9sz0 with job batch system ID: 2 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:32.501322653Z [2020-10-03T15:33:32+0000] [MainThread] [I] [toil.leader] Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_get_submission.cwl/instance-uiqcdfx0 with job batch system ID: 3 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:32.501522573Z [2020-10-03T15:33:32+0000] [MainThread] [I] [toil.leader] Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_get_docker_config.cwl/instance-wwx9ulj7 with job batch system ID: 4 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:32.501704883Z [2020-10-03T15:33:32+0000] [MainThread] [I] [toil.leader] Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_set_permissions.cwl/instance-iort46mr with job batch system ID: 5 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:33.536382136Z [2020-10-03T15:33:33+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmpux27maqk/worker_log.txt STDERR: 2020-10-03T15:33:33.579116792Z [2020-10-03T15:33:33+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmpm971_vxg/worker_log.txt STDERR: 2020-10-03T15:33:33.586183691Z [2020-10-03T15:33:33+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmpb9ueqs0g/worker_log.txt STDERR: 2020-10-03T15:33:33.597050365Z [2020-10-03T15:33:33+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmptdsbhqm6/worker_log.txt STDERR: 2020-10-03T15:33:33.625597843Z [2020-10-03T15:33:33+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmpqybhp_w2/worker_log.txt STDERR: 2020-10-03T15:33:33.798859189Z [2020-10-03T15:33:33+0000] [MainThread] [I] [toil.leader] Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/create_adminsynid_annotation.cwl' echo kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_create_adminsynid_annotation.cwl/instance-pkfi9sz0 STDERR: 2020-10-03T15:33:33.799767900Z [2020-10-03T15:33:33+0000] [MainThread] [I] [toil.leader] Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_annotate_submission.cwl/instance-e1j0qgls with job batch system ID: 6 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:34.835436741Z [2020-10-03T15:33:34+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmpd60o2cs9/worker_log.txt STDERR: 2020-10-03T15:33:36.802940990Z [2020-10-03T15:33:36+0000] [MainThread] [I] [toil.leader] Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_set_permissions.cwl/instance-g547d1ny STDERR: 2020-10-03T15:33:36.863581858Z [2020-10-03T15:33:36+0000] [MainThread] [I] [toil.leader] Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_set_permissions.cwl/instance-iort46mr STDERR: 2020-10-03T15:33:37.076921495Z [2020-10-03T15:33:37+0000] [MainThread] [I] [toil.leader] Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_get_docker_config.cwl/instance-wwx9ulj7 STDERR: 2020-10-03T15:33:37.754421500Z [2020-10-03T15:33:37+0000] [MainThread] [I] [toil.leader] Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_get_submission.cwl/instance-uiqcdfx0 STDERR: 2020-10-03T15:33:37.755227375Z [2020-10-03T15:33:37+0000] [MainThread] [I] [toil.leader] Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_evaluation_id.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_evaluation_id.cwl/instance-o7v_yaac with job batch system ID: 7 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:37.755352864Z [2020-10-03T15:33:37+0000] [MainThread] [I] [toil.leader] Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r with job batch system ID: 8 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:37.755504693Z [2020-10-03T15:33:37+0000] [MainThread] [I] [toil.leader] Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_dataset.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_dataset.cwl/instance-mpftthlb with job batch system ID: 9 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:38.775308370Z [2020-10-03T15:33:38+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmpul_5vaqo/worker_log.txt STDERR: 2020-10-03T15:33:38.785261921Z [2020-10-03T15:33:38+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/worker_log.txt STDERR: 2020-10-03T15:33:38.812346130Z [2020-10-03T15:33:38+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp7yh0vgyx/worker_log.txt STDERR: 2020-10-03T15:33:38.826842634Z [2020-10-03T15:33:38+0000] [MainThread] [I] [toil.leader] Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_annotate_submission.cwl/instance-e1j0qgls STDERR: 2020-10-03T15:33:38.827301912Z [2020-10-03T15:33:38+0000] [MainThread] [I] [toil.leader] Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/create_adminsynid_annotation.cwl' echo kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_create_adminsynid_annotation.cwl/instance-pkfi9sz0 with job batch system ID: 10 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:39.818366515Z [2020-10-03T15:33:39+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp20_6l54z/worker_log.txt STDERR: 2020-10-03T15:33:39.947538179Z [2020-10-03T15:33:39+0000] [MainThread] [I] [toil.leader] Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/create_adminsynid_annotation.cwl' echo kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_create_adminsynid_annotation.cwl/instance-pkfi9sz0 STDERR: 2020-10-03T15:33:40.836457472Z [2020-10-03T15:33:40+0000] [Thread-122] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl file:/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/tmp4n6tnd_5 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r. STDERR: 2020-10-03T15:33:40.836916199Z [2020-10-03T15:33:40+0000] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r STDERR: 2020-10-03T15:33:40.838188802Z [2020-10-03T15:33:40+0000] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r STDERR: 2020-10-03T15:33:40.838600249Z [2020-10-03T15:33:40+0000] [MainThread] [W] [toil.leader] Log from job kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r follows: STDERR: 2020-10-03T15:33:40.838612409Z =========> STDERR: 2020-10-03T15:33:40.838618429Z [2020-10-03T15:33:38+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- STDERR: 2020-10-03T15:33:40.838624599Z [2020-10-03T15:33:38+0000] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host 4207e15d3a9b. STDERR: 2020-10-03T15:33:40.838630539Z [2020-10-03T15:33:38+0000] [MainThread] [W] [cwltool] [job validate_docker.cwl] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance. STDERR: 2020-10-03T15:33:40.838650199Z [2020-10-03T15:33:38+0000] [MainThread] [I] [cwltool] [job validate_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/tmxn0a464/tmp-out988d1zk7$ docker \ STDERR: 2020-10-03T15:33:40.838654549Z run \ STDERR: 2020-10-03T15:33:40.838657659Z -i \ STDERR: 2020-10-03T15:33:40.838660889Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/tmxn0a464/tmp-out988d1zk7,target=/FmlgYK \ STDERR: 2020-10-03T15:33:40.838664589Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/t7jg2rovfzzdl4fq6,target=/tmp \ STDERR: 2020-10-03T15:33:40.838668959Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/tmp_r4cpe9e.tmp,target=/var/lib/cwl/stg35ebb40b-63ff-4a5b-83d6-bf99221b8d2c/.synapseConfig,readonly \ STDERR: 2020-10-03T15:33:40.838672589Z --workdir=/FmlgYK \ STDERR: 2020-10-03T15:33:40.838675549Z --read-only=true \ STDERR: 2020-10-03T15:33:40.838678469Z --user=0:0 \ STDERR: 2020-10-03T15:33:40.838681379Z --rm \ STDERR: 2020-10-03T15:33:40.838684259Z --env=TMPDIR=/tmp \ STDERR: 2020-10-03T15:33:40.838687419Z --env=HOME=/FmlgYK \ STDERR: 2020-10-03T15:33:40.838690349Z --cidfile=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/t7jg2rovfh5ax3hw4/20201003153338-874132.cid \ STDERR: 2020-10-03T15:33:40.838694119Z sagebionetworks/synapsepythonclient:v2.1.0 \ STDERR: 2020-10-03T15:33:40.838697079Z python3 \ STDERR: 2020-10-03T15:33:40.838700069Z validate_docker.py \ STDERR: 2020-10-03T15:33:40.838702968Z -p \ STDERR: 2020-10-03T15:33:40.838705998Z docker.synapse.org/syn22920709/example1_test \ STDERR: 2020-10-03T15:33:40.838708928Z -d \ STDERR: 2020-10-03T15:33:40.838711858Z sha256:c2cb9dcba2819a1ca3e7be8da49419b1ede7c6f0357c4f04fdb3ba5c5dd78c00 \ STDERR: 2020-10-03T15:33:40.838715058Z -c \ STDERR: 2020-10-03T15:33:40.838718028Z /var/lib/cwl/stg35ebb40b-63ff-4a5b-83d6-bf99221b8d2c/.synapseConfig \ STDERR: 2020-10-03T15:33:40.838721068Z -r \ STDERR: 2020-10-03T15:33:40.838723908Z results.json STDERR: 2020-10-03T15:33:40.838726918Z Traceback (most recent call last): STDERR: 2020-10-03T15:33:40.838734528Z File "validate_docker.py", line 65, in STDERR: 2020-10-03T15:33:40.838739038Z labels = blob_resp.json()['container_config']['Labels'] STDERR: 2020-10-03T15:33:40.838742008Z KeyError: 'container_config' STDERR: 2020-10-03T15:33:40.838744898Z [2020-10-03T15:33:40+0000] [MainThread] [I] [cwltool] [job validate_docker.cwl] Max memory used: 0MiB STDERR: 2020-10-03T15:33:40.838747968Z [2020-10-03T15:33:40+0000] [MainThread] [E] [cwltool] [job validate_docker.cwl] Job error: STDERR: 2020-10-03T15:33:40.838751028Z ("Error collecting output for parameter 'results':\ncovid19-challenge-master/infrastructure/validate_docker.cwl:163:7: Did not find output file with glob pattern: '['results.json']'", {}) STDERR: 2020-10-03T15:33:40.838754668Z [2020-10-03T15:33:40+0000] [MainThread] [W] [cwltool] [job validate_docker.cwl] completed permanentFail STDERR: 2020-10-03T15:33:40.838757878Z Traceback (most recent call last): STDERR: 2020-10-03T15:33:40.838761068Z File "/usr/local/lib/python3.6/site-packages/toil/worker.py", line 368, in workerScript STDERR: 2020-10-03T15:33:40.838764398Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) STDERR: 2020-10-03T15:33:40.838767598Z File "/usr/local/lib/python3.6/site-packages/toil/job.py", line 1424, in _runner STDERR: 2020-10-03T15:33:40.838770848Z returnValues = self._run(jobGraph, fileStore) STDERR: 2020-10-03T15:33:40.838773768Z File "/usr/local/lib/python3.6/site-packages/toil/job.py", line 1361, in _run STDERR: 2020-10-03T15:33:40.838776868Z return self.run(fileStore) STDERR: 2020-10-03T15:33:40.838779748Z File "/usr/local/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 988, in run STDERR: 2020-10-03T15:33:40.838783268Z raise cwltool.errors.WorkflowException(status) STDERR: 2020-10-03T15:33:40.838786188Z cwltool.errors.WorkflowException: permanentFail STDERR: 2020-10-03T15:33:40.838789188Z [2020-10-03T15:33:40+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 4207e15d3a9b STDERR: 2020-10-03T15:33:40.838792298Z <========= STDERR: 2020-10-03T15:33:40.840704147Z [2020-10-03T15:33:40+0000] [MainThread] [W] [toil.jobGraph] Due to failure we are reducing the remaining retry count of job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r to 0 STDERR: 2020-10-03T15:33:40.846059625Z [2020-10-03T15:33:40+0000] [MainThread] [W] [toil.leader] Job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r is completely failed STDERR: 2020-10-03T15:33:42.623549255Z [2020-10-03T15:33:42+0000] [MainThread] [I] [toil.leader] Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_evaluation_id.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_evaluation_id.cwl/instance-o7v_yaac STDERR: 2020-10-03T15:33:42.785191734Z [2020-10-03T15:33:42+0000] [MainThread] [I] [toil.leader] Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_dataset.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_dataset.cwl/instance-mpftthlb STDERR: 2020-10-03T15:33:42.785862160Z [2020-10-03T15:33:42+0000] [MainThread] [I] [toil.leader] Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/dockstore-tool-synapse/v0.2/cwl/synapse-get-tool.cwl' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_dockstore-tool-synapse_v0.2_cwl_synapse-get-tool.cwl/instance-1lh4sp9g with job batch system ID: 11 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:42.786023829Z [2020-10-03T15:33:42+0000] [MainThread] [I] [toil.leader] Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_annotate_submission.cwl/instance-c2d_lvrj with job batch system ID: 12 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2020-10-03T15:33:43.778840217Z [2020-10-03T15:33:43+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmpzjt93w06/worker_log.txt STDERR: 2020-10-03T15:33:43.781194693Z [2020-10-03T15:33:43+0000] [MainThread] [I] [toil.worker] Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp6j1197bh/worker_log.txt STDERR: 2020-10-03T15:33:45.869209523Z [2020-10-03T15:33:45+0000] [MainThread] [I] [toil.leader] Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/dockstore-tool-synapse/v0.2/cwl/synapse-get-tool.cwl' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_dockstore-tool-synapse_v0.2_cwl_synapse-get-tool.cwl/instance-1lh4sp9g STDERR: 2020-10-03T15:33:47.503587232Z [2020-10-03T15:33:47+0000] [MainThread] [I] [toil.leader] Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_annotate_submission.cwl/instance-c2d_lvrj STDERR: 2020-10-03T15:33:57.109914740Z [2020-10-03T15:33:57+0000] [MainThread] [I] [toil.leader] Finished toil run with 7 failed jobs. STDERR: 2020-10-03T15:33:57.110286658Z [2020-10-03T15:33:57+0000] [MainThread] [I] [toil.leader] Failed jobs at end of the run: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_evaluation_id.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_evaluation_id.cwl/instance-o7v_yaac 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/dockstore-tool-synapse/v0.2/cwl/synapse-get-tool.cwl' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_dockstore-tool-synapse_v0.2_cwl_synapse-get-tool.cwl/instance-1lh4sp9g 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_get_submission.cwl/instance-uiqcdfx0 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_get_docker_config.cwl/instance-wwx9ulj7 'CWLWorkflow' kind-CWLWorkflow/instance-pv636rz9 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_dataset.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_dataset.cwl/instance-mpftthlb STDERR: 2020-10-03T15:33:57.122875096Z Traceback (most recent call last): STDERR: 2020-10-03T15:33:57.122888976Z File "/usr/local/bin/toil-cwl-runner", line 8, in STDERR: 2020-10-03T15:33:57.122893376Z sys.exit(main()) STDERR: 2020-10-03T15:33:57.122896956Z File "/usr/local/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 1788, in main STDERR: 2020-10-03T15:33:57.122900576Z outobj = toil.start(wf1) STDERR: 2020-10-03T15:33:57.122903696Z File "/usr/local/lib/python3.6/site-packages/toil/common.py", line 846, in start STDERR: 2020-10-03T15:33:57.122907066Z return self._runMainLoop(rootJobGraph) STDERR: 2020-10-03T15:33:57.122910336Z File "/usr/local/lib/python3.6/site-packages/toil/common.py", line 1138, in _runMainLoop STDERR: 2020-10-03T15:33:57.122913526Z jobCache=self._jobCache).run() STDERR: 2020-10-03T15:33:57.122916386Z File "/usr/local/lib/python3.6/site-packages/toil/leader.py", line 269, in run STDERR: 2020-10-03T15:33:57.122919476Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore) STDERR: 2020-10-03T15:33:57.122923586Z toil.leader.FailedJobsException: The job store 'file:/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/tmp4n6tnd_5' contains 7 failed jobs: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_evaluation_id.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_evaluation_id.cwl/instance-o7v_yaac, 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/dockstore-tool-synapse/v0.2/cwl/synapse-get-tool.cwl' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_dockstore-tool-synapse_v0.2_cwl_synapse-get-tool.cwl/instance-1lh4sp9g, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_get_submission.cwl/instance-uiqcdfx0, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v2.7/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v2.7_get_docker_config.cwl/instance-wwx9ulj7, 'CWLWorkflow' kind-CWLWorkflow/instance-pv636rz9, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/get_dataset.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_get_dataset.cwl/instance-mpftthlb STDERR: 2020-10-03T15:33:57.122937876Z Log from job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/covid19-challenge-master/infrastructure/validate_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_2b8dbdc5-f219-4ed9-83e0-2748e994de6c_covid19-challenge-master_infrastructure_validate_docker.cwl/instance-m_jc_9_r follows: STDERR: 2020-10-03T15:33:57.122941746Z =========> STDERR: 2020-10-03T15:33:57.122944736Z [2020-10-03T15:33:38+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- STDERR: 2020-10-03T15:33:57.122948256Z [2020-10-03T15:33:38+0000] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host 4207e15d3a9b. STDERR: 2020-10-03T15:33:57.122951616Z [2020-10-03T15:33:38+0000] [MainThread] [W] [cwltool] [job validate_docker.cwl] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance. STDERR: 2020-10-03T15:33:57.122955116Z [2020-10-03T15:33:38+0000] [MainThread] [I] [cwltool] [job validate_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/tmxn0a464/tmp-out988d1zk7$ docker \ STDERR: 2020-10-03T15:33:57.122958796Z run \ STDERR: 2020-10-03T15:33:57.122961846Z -i \ STDERR: 2020-10-03T15:33:57.122965486Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/tmxn0a464/tmp-out988d1zk7,target=/FmlgYK \ STDERR: 2020-10-03T15:33:57.122969236Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/t7jg2rovfzzdl4fq6,target=/tmp \ STDERR: 2020-10-03T15:33:57.122972786Z --mount=type=bind,source=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/tmp_r4cpe9e.tmp,target=/var/lib/cwl/stg35ebb40b-63ff-4a5b-83d6-bf99221b8d2c/.synapseConfig,readonly \ STDERR: 2020-10-03T15:33:57.122979686Z --workdir=/FmlgYK \ STDERR: 2020-10-03T15:33:57.122982896Z --read-only=true \ STDERR: 2020-10-03T15:33:57.122985826Z --user=0:0 \ STDERR: 2020-10-03T15:33:57.122988766Z --rm \ STDERR: 2020-10-03T15:33:57.122991656Z --env=TMPDIR=/tmp \ STDERR: 2020-10-03T15:33:57.122994626Z --env=HOME=/FmlgYK \ STDERR: 2020-10-03T15:33:57.122997536Z --cidfile=/var/lib/docker/volumes/workflow_orchestrator_shared/_data/2b8dbdc5-f219-4ed9-83e0-2748e994de6c/node-5b111b6b-d3c2-4d93-850d-6ea815b3cd82-c31eea55a29431535ff01de94bdcf5cf/tmp62a7_3_g/fb5e9dc9-48e7-42f0-8a3b-d4f95a0ef5bd/t7jg2rovfh5ax3hw4/20201003153338-874132.cid \ STDERR: 2020-10-03T15:33:57.123000946Z sagebionetworks/synapsepythonclient:v2.1.0 \ STDERR: 2020-10-03T15:33:57.123003916Z python3 \ STDERR: 2020-10-03T15:33:57.123006756Z validate_docker.py \ STDERR: 2020-10-03T15:33:57.123009636Z -p \ STDERR: 2020-10-03T15:33:57.123012506Z docker.synapse.org/syn22920709/example1_test \ STDERR: 2020-10-03T15:33:57.123015725Z -d \ STDERR: 2020-10-03T15:33:57.123018845Z sha256:c2cb9dcba2819a1ca3e7be8da49419b1ede7c6f0357c4f04fdb3ba5c5dd78c00 \ STDERR: 2020-10-03T15:33:57.123022125Z -c \ STDERR: 2020-10-03T15:33:57.123025165Z /var/lib/cwl/stg35ebb40b-63ff-4a5b-83d6-bf99221b8d2c/.synapseConfig \ STDERR: 2020-10-03T15:33:57.123028285Z -r \ STDERR: 2020-10-03T15:33:57.123031115Z results.json STDERR: 2020-10-03T15:33:57.123034045Z Traceback (most recent call last): STDERR: 2020-10-03T15:33:57.123037275Z File "validate_docker.py", line 65, in STDERR: 2020-10-03T15:33:57.123040595Z labels = blob_resp.json()['container_config']['Labels'] STDERR: 2020-10-03T15:33:57.123043545Z KeyError: 'container_config' STDERR: 2020-10-03T15:33:57.123046445Z [2020-10-03T15:33:40+0000] [MainThread] [I] [cwltool] [job validate_docker.cwl] Max memory used: 0MiB STDERR: 2020-10-03T15:33:57.123049515Z [2020-10-03T15:33:40+0000] [MainThread] [E] [cwltool] [job validate_docker.cwl] Job error: STDERR: 2020-10-03T15:33:57.123053145Z ("Error collecting output for parameter 'results':\ncovid19-challenge-master/infrastructure/validate_docker.cwl:163:7: Did not find output file with glob pattern: '['results.json']'", {}) STDERR: 2020-10-03T15:33:57.123056665Z [2020-10-03T15:33:40+0000] [MainThread] [W] [cwltool] [job validate_docker.cwl] completed permanentFail STDERR: 2020-10-03T15:33:57.123059725Z Traceback (most recent call last): STDERR: 2020-10-03T15:33:57.123062655Z File "/usr/local/lib/python3.6/site-packages/toil/worker.py", line 368, in workerScript STDERR: 2020-10-03T15:33:57.123065805Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) STDERR: 2020-10-03T15:33:57.123068975Z File "/usr/local/lib/python3.6/site-packages/toil/job.py", line 1424, in _runner STDERR: 2020-10-03T15:33:57.123076175Z returnValues = self._run(jobGraph, fileStore) STDERR: 2020-10-03T15:33:57.123079605Z File "/usr/local/lib/python3.6/site-packages/toil/job.py", line 1361, in _run STDERR: 2020-10-03T15:33:57.123082885Z return self.run(fileStore) STDERR: 2020-10-03T15:33:57.123085755Z File "/usr/local/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 988, in run STDERR: 2020-10-03T15:33:57.123088855Z raise cwltool.errors.WorkflowException(status) STDERR: 2020-10-03T15:33:57.123091745Z cwltool.errors.WorkflowException: permanentFail STDERR: 2020-10-03T15:33:57.123094665Z [2020-10-03T15:33:40+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 4207e15d3a9b STDERR: 2020-10-03T15:33:57.123097685Z <========= ```

Created by Jonathan Stubblefield ShadowUnicorn90
Hi, I noticed that others have submitted successfully recently, so I tried to submit the Question1 example again. Unfortunately I got the same result as before: STDERR: 2021-02-10T18:07:58.167849630Z ERROR:cwltool:[job validate_docker.cwl] Job error: STDERR: 2021-02-10T18:07:58.167854590Z ("Error collecting output for parameter 'results':\ncovid19-challenge-master/infrastructure/validate_docker.cwl:163:7: Did not find output file with glob pattern: '['results.json']'", {}) In case it matters, I am submitting from a Windows 10 PC with recently installed Docker version 20.10.2, build 2291f61 I don't see any references to validate_docker.cwl or results.json in the resources checked out from git, so I'm not sure what to check. Please let me know if there is something I can do on my end. Thanks, Jeff
Hi @jeff_green, We're currently working on fixing this issue. Thank you for your patience, @trberg
I received the same error when submitting the q2 example (docker.synapse.org/syn24262913/awesome-covid19-q2-model): STDERR: 2021-02-04T19:23:42.405426363Z KeyError: 'container_config' STDERR: 2021-02-04T19:23:42.405430883Z [job validate_docker.cwl] Max memory used: 0MiB STDERR: 2021-02-04T19:23:42.405435283Z INFO:cwltool:[job validate_docker.cwl] Max memory used: 0MiB STDERR: 2021-02-04T19:23:42.405439763Z [job validate_docker.cwl] Job error: STDERR: 2021-02-04T19:23:42.405444063Z ("Error collecting output for parameter 'results':\ncovid19-challenge-master/infrastructure/validate_docker.cwl:163:7: Did not find output file with glob pattern: '['results.json']'", {}) STDERR: 2021-02-04T19:23:42.405455663Z ERROR:cwltool:[job validate_docker.cwl] Job error: STDERR: 2021-02-04T19:23:42.405461213Z ("Error collecting output for parameter 'results':\ncovid19-challenge-master/infrastructure/validate_docker.cwl:163:7: Did not find output file with glob pattern: '['results.json']'", {}) STDERR: 2021-02-04T19:23:42.405466583Z [job validate_docker.cwl] completed permanentFail
Hi, The same error occurs for me when trying to submit the example solution for question 1. The submission was: docker.synapse.org/syn24262913/awesome-covid19-q1-model The relevant lines from the log are below. Do you know the solution? Thank you, Jeff The error in the log: STDERR: 2021-02-04T15:53:33.385652793Z KeyError: 'container_config' STDERR: 2021-02-04T15:53:33.385655843Z [job validate_docker.cwl] Max memory used: 0MiB STDERR: 2021-02-04T15:53:33.385658883Z INFO:cwltool:[job validate_docker.cwl] Max memory used: 0MiB STDERR: 2021-02-04T15:53:33.385662003Z [job validate_docker.cwl] Job error: STDERR: 2021-02-04T15:53:33.385665043Z ("Error collecting output for parameter 'results':\ncovid19-challenge-master/infrastructure/validate_docker.cwl:163:7: Did not find output file with glob pattern: '['results.json']'", {}) STDERR: 2021-02-04T15:53:33.385672193Z ERROR:cwltool:[job validate_docker.cwl] Job error: STDERR: 2021-02-04T15:53:33.385675433Z ("Error collecting output for parameter 'results':\ncovid19-challenge-master/infrastructure/validate_docker.cwl:163:7: Did not find output file with glob pattern: '['results.json']'", {}) STDERR: 2021-02-04T15:53:33.385679043Z [job validate_docker.cwl] completed permanentFail
Yes, my team seems to be doing better than me. I still have issues with submission.
Hi @ShadowUnicorn90 , It looks like your team had a valid submission, are you still having issues here? Thanks, Tim
They still fail for me.
Hi @ShadowUnicorn90, We recently updated the data and were able to run our example models for both questions. Can you try and submit those again? Let us know if they are still failing. Thank you, @trberg
My submissions were docker.synapse.org/syn22920709/jws_dream_q1 and docker.synapse.org/syn22920709/example1_test. The second is your code but running as a test.
Hi @ShadowUnicorn90 , Which submission is your's? We fixed the issues that were causing failures for previous submissions and were able to get the test models to run. I'll look into your submission further, Thank you, Tim
Hello! I wanted to check in on this issue. I haven't seen any discussion, and as recently as 10/9, the system for question 1 still does not seem to be working.
Hi @ShadowUnicorn90 , We're looking into the issue, Thank you, @trberg

Example Code For Question 1 Not Working page is loading…