STDERR: 2019-10-11T09:02:16.845808286Z WARNING:toil.leader:O/e/jobgAbzCO _with_retry(lambda: annotate_submission(syn, args.submissionid, args.annotation_values, to_public=args.to_public, force_change_annotation_acl=args.force_change_annotation_acl),wait=3,retries=10) STDERR: 2019-10-11T09:02:16.845894567Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/retry.py", line 105, in _with_retry STDERR: 2019-10-11T09:02:16.845998278Z WARNING:toil.leader:O/e/jobgAbzCO six.reraise(exc_info[0], exc_info[1], exc_info[2]) STDERR: 2019-10-11T09:02:16.846190547Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/retry.py", line 53, in _with_retry STDERR: 2019-10-11T09:02:16.846350139Z WARNING:toil.leader:O/e/jobgAbzCO response = function() STDERR: 2019-10-11T09:02:16.846442638Z WARNING:toil.leader:O/e/jobgAbzCO File "annotationSubmission.py", line 75, in STDERR: 2019-10-11T09:02:16.846616679Z WARNING:toil.leader:O/e/jobgAbzCO _with_retry(lambda: annotate_submission(syn, args.submissionid, args.annotation_values, to_public=args.to_public, force_change_annotation_acl=args.force_change_annotation_acl),wait=3,retries=10) STDERR: 2019-10-11T09:02:16.846798920Z WARNING:toil.leader:O/e/jobgAbzCO File "annotationSubmission.py", line 61, in annotate_submission STDERR: 2019-10-11T09:02:16.846976460Z WARNING:toil.leader:O/e/jobgAbzCO status = syn.store(status) STDERR: 2019-10-11T09:02:16.847180662Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/client.py", line 912, in store STDERR: 2019-10-11T09:02:16.847238912Z WARNING:toil.leader:O/e/jobgAbzCO return type(obj)(**self.restPUT(obj.putURI(), obj.json())) STDERR: 2019-10-11T09:02:16.847326752Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/client.py", line 3368, in restPUT STDERR: 2019-10-11T09:02:16.847408072Z WARNING:toil.leader:O/e/jobgAbzCO exceptions._raise_for_status(response, verbose=self.debug) STDERR: 2019-10-11T09:02:16.847547082Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/exceptions.py", line 153, in _raise_for_status STDERR: 2019-10-11T09:02:16.847606193Z WARNING:toil.leader:O/e/jobgAbzCO raise SynapseHTTPError(message, response=response) STDERR: 2019-10-11T09:02:16.847712163Z WARNING:toil.leader:O/e/jobgAbzCO synapseclient.exceptions.SynapseHTTPError: 412 Client Error: STDERR: 2019-10-11T09:02:16.847793554Z WARNING:toil.leader:O/e/jobgAbzCO Submission Status: 9693830 was updated since you last fetched it, retrieve it again and reapply the update STDERR: 2019-10-11T09:02:16.847895994Z WARNING:toil.leader:O/e/jobgAbzCO [job annotate_submission.cwl] Max memory used: 45MiB STDERR: 2019-10-11T09:02:16.847978974Z WARNING:toil.leader:O/e/jobgAbzCO INFO:cwltool:[job annotate_submission.cwl] Max memory used: 45MiB STDERR: 2019-10-11T09:02:16.848032944Z WARNING:toil.leader:O/e/jobgAbzCO [job annotate_submission.cwl] completed permanentFail STDERR: 2019-10-11T09:02:16.848116075Z WARNING:toil.leader:O/e/jobgAbzCO WARNING:cwltool:[job annotate_submission.cwl] completed permanentFail STDERR: 2019-10-11T09:02:16.848181275Z WARNING:toil.leader:O/e/jobgAbzCO Traceback (most recent call last): STDERR: 2019-10-11T09:02:16.848264375Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/toil/worker.py", line 331, in workerScript STDERR: 2019-10-11T09:02:16.848323155Z WARNING:toil.leader:O/e/jobgAbzCO job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) STDERR: 2019-10-11T09:02:16.848424796Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1378, in _runner STDERR: 2019-10-11T09:02:16.848521136Z WARNING:toil.leader:O/e/jobgAbzCO returnValues = self._run(jobGraph, fileStore) STDERR: 2019-10-11T09:02:16.848617087Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1323, in _run STDERR: 2019-10-11T09:02:16.848681536Z WARNING:toil.leader:O/e/jobgAbzCO return self.run(fileStore) STDERR: 2019-10-11T09:02:16.848783537Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 606, in run STDERR: 2019-10-11T09:02:16.848835347Z WARNING:toil.leader:O/e/jobgAbzCO raise cwltool.errors.WorkflowException(status) STDERR: 2019-10-11T09:02:16.848921158Z WARNING:toil.leader:O/e/jobgAbzCO WorkflowException: permanentFail STDERR: 2019-10-11T09:02:16.848977148Z WARNING:toil.leader:O/e/jobgAbzCO ERROR:toil.worker:Exiting the worker because of a failed job on host fb0a4e18f2b6 STDERR: 2019-10-11T09:02:16.849137638Z WARNING:toil.leader:O/e/jobgAbzCO WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO with ID O/e/jobgAbzCO to 0 STDERR: 2019-10-11T09:02:16.849974372Z WARNING:toil.leader:Job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO with ID O/e/jobgAbzCO is completely failed STDERR: 2019-10-11T09:02:17.536308186Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_email.cwl' python f/S/jobW0HJQ7 STDERR: 2019-10-11T09:02:17.649865633Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:02:17.651042818Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpsUaR_v/worker_log.txt STDERR: 2019-10-11T09:02:21.260331349Z INFO:toil.leader:Job ended successfully: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/submit_to_challenge.cwl' python D/t/jobBT4ymB STDERR: 2019-10-11T09:02:21.260517391Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python 9/8/jobMrBtrM with job batch system ID: 15 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T09:02:21.947581207Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:02:21.948908681Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpg46MYY/worker_log.txt STDERR: 2019-10-11T09:02:22.039889007Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python 9/8/jobMrBtrM STDERR: 2019-10-11T09:02:22.042349566Z INFO:toil.leader:Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/validate.cwl' python G/0/jobzBCSc4 with job batch system ID: 16 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T09:02:22.666455150Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:02:22.667485404Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpHfx4e1/worker_log.txt STDERR: 2019-10-11T09:02:22.770922494Z INFO:toil.leader:Job ended successfully: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/validate.cwl' python G/0/jobzBCSc4 STDERR: 2019-10-11T09:02:22.773802695Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python p/7/job9mPjVw with job batch system ID: 17 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T09:02:23.389965881Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:02:23.391183415Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmp9Ei3aR/worker_log.txt STDERR: 2019-10-11T09:02:23.499023062Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python p/7/job9mPjVw STDERR: 2019-10-11T09:02:33.595430779Z INFO:toil.leader:Finished toil run with 8 failed jobs. STDERR: 2019-10-11T09:02:33.596009471Z INFO:toil.leader:Failed jobs at end of the run: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_training_docker.cwl' python C/y/job4RYdld 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python R/t/jobUZXriU 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python m/p/jobcphbSj 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/upload_to_synapse.cwl' python j/I/jobrBWwTu 'CWLWorkflow' 8/U/jobniYcvf 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_infer_docker.cwl' python E/S/jobt6lgEA 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_docker.cwl' python C/h/job3b7SVR 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO STDERR: 2019-10-11T09:02:33.683849084Z Traceback (most recent call last): STDERR: 2019-10-11T09:02:33.683877244Z File "/usr/local/bin/toil-cwl-runner", line 10, in STDERR: 2019-10-11T09:02:33.683882544Z sys.exit(main()) STDERR: 2019-10-11T09:02:33.683886614Z File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 1276, in main STDERR: 2019-10-11T09:02:33.684134945Z outobj = toil.start(wf1) STDERR: 2019-10-11T09:02:33.684176775Z File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 781, in start STDERR: 2019-10-11T09:02:33.684298716Z return self._runMainLoop(rootJobGraph) STDERR: 2019-10-11T09:02:33.684308816Z File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 1054, in _runMainLoop STDERR: 2019-10-11T09:02:33.684594787Z jobCache=self._jobCache).run() STDERR: 2019-10-11T09:02:33.684607187Z File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 246, in run STDERR: 2019-10-11T09:02:33.684611617Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore) STDERR: 2019-10-11T09:02:33.684632247Z toil.leader.FailedJobsException " />

I submitted my image which completes under Fast Lane. In the challenge submission it fails with the following log. Any suggestion greatly appreciated. I think the point of interest is: STDERR: 2019-10-11T09:02:16.847606193Z WARNING:toil.leader:O/e/jobgAbzCO raise SynapseHTTPError(message, response=response) STDERR: 2019-10-11T09:02:16.847712163Z WARNING:toil.leader:O/e/jobgAbzCO synapseclient.exceptions.SynapseHTTPError: 412 Client Error: FULL: STDERR: 2019-10-11T08:56:49.589826204Z INFO:cwltool:Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/main_docker_agent_workflow.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/main_docker_agent_workflow.cwl' STDERR: 2019-10-11T08:56:52.800387270Z EHR-challenge-develop/main_docker_agent_workflow.cwl:143:9: object id `EHR-challenge-develop/main_docker_agent_workflow.cwl#run_docker_infer/status` previously defined STDERR: 2019-10-11T08:56:52.800440230Z WARNING:salad:EHR-challenge-develop/main_docker_agent_workflow.cwl:143:9: object id `EHR-challenge-develop/main_docker_agent_workflow.cwl#run_docker_infer/status` previously defined STDERR: 2019-10-11T08:56:52.802392148Z EHR-challenge-develop/main_docker_agent_workflow.cwl:117:9: object id `EHR-challenge-develop/main_docker_agent_workflow.cwl#run_docker_train/status` previously defined STDERR: 2019-10-11T08:56:52.802411678Z WARNING:salad:EHR-challenge-develop/main_docker_agent_workflow.cwl:117:9: object id `EHR-challenge-develop/main_docker_agent_workflow.cwl#run_docker_train/status` previously defined STDERR: 2019-10-11T08:56:59.621498847Z EHR-challenge-develop/run_synthetic_infer_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_infer_docker.cwl#status` previously defined STDERR: 2019-10-11T08:56:59.621561207Z WARNING:salad:EHR-challenge-develop/run_synthetic_infer_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_infer_docker.cwl#status` previously defined STDERR: 2019-10-11T08:57:00.041943295Z EHR-challenge-develop/run_synthetic_training_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_training_docker.cwl#status` previously defined STDERR: 2019-10-11T08:57:00.042004175Z WARNING:salad:EHR-challenge-develop/run_synthetic_training_docker.cwl:22:5: object id `EHR-challenge-develop/run_synthetic_training_docker.cwl#status` previously defined STDERR: 2019-10-11T08:57:02.255005598Z WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (32). STDERR: 2019-10-11T08:57:02.255192520Z WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (268107968512). STDERR: 2019-10-11T08:57:02.255621361Z WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (327929716736). STDERR: 2019-10-11T08:57:02.851613973Z INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932. STDERR: 2019-10-11T08:57:03.728129747Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T08:57:03.729754794Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpHiy6fw/worker_log.txt STDERR: 2019-10-11T08:57:05.280798494Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python p/7/job9mPjVw with job batch system ID: 1 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T08:57:05.280965925Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python m/p/jobcphbSj with job batch system ID: 2 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T08:57:05.304734338Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python R/t/jobUZXriU with job batch system ID: 3 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T08:57:05.307992350Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/notification_email.cwl' python O/X/jobIXelLz with job batch system ID: 4 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T08:57:06.100644318Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T08:57:06.105372645Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpCtQdF5/worker_log.txt STDERR: 2019-10-11T08:57:06.139646619Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T08:57:06.143353544Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpyaTeoM/worker_log.txt STDERR: 2019-10-11T08:57:06.163606213Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T08:57:06.165244169Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmp55a8YL/worker_log.txt STDERR: 2019-10-11T08:57:06.218745927Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T08:57:06.220020242Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpsXChqm/worker_log.txt STDERR: 2019-10-11T08:57:09.035478393Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python p/7/job9mPjVw STDERR: 2019-10-11T08:57:09.978791094Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/notification_email.cwl' python O/X/jobIXelLz STDERR: 2019-10-11T08:57:10.465191488Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python m/p/jobcphbSj STDERR: 2019-10-11T08:57:10.794114027Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python R/t/jobUZXriU STDERR: 2019-10-11T08:57:10.794893300Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_docker.cwl' python C/h/job3b7SVR with job batch system ID: 5 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T08:57:11.647407708Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T08:57:11.648778954Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmppiG6Z4/worker_log.txt STDERR: 2019-10-11T08:57:16.251194536Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_docker.cwl' python C/h/job3b7SVR STDERR: 2019-10-11T08:57:16.252474871Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python u/e/job8rtC9_ with job batch system ID: 6 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T08:57:16.263066802Z INFO:toil.leader:Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_training_docker.cwl' python C/y/job4RYdld with job batch system ID: 7 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T08:57:16.938142648Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T08:57:16.939199872Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpf8PP8t/worker_log.txt STDERR: 2019-10-11T08:57:16.955715166Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T08:57:16.957930995Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpLHEfdm/worker_log.txt STDERR: 2019-10-11T08:57:20.905365029Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python u/e/job8rtC9_ STDERR: 2019-10-11T09:00:10.348889645Z INFO:toil.leader:Job ended successfully: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_training_docker.cwl' python C/y/job4RYdld STDERR: 2019-10-11T09:00:10.349860799Z INFO:toil.leader:Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_infer_docker.cwl' python E/S/jobt6lgEA with job batch system ID: 8 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T09:00:10.921143657Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:00:10.922415522Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpqkJB5C/worker_log.txt STDERR: 2019-10-11T09:02:05.428061251Z INFO:toil.leader:Job ended successfully: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_infer_docker.cwl' python E/S/jobt6lgEA STDERR: 2019-10-11T09:02:05.429394346Z INFO:toil.leader:Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/validate.cwl' python G/0/jobzBCSc4 with job batch system ID: 9 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T09:02:05.445570156Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/upload_to_synapse.cwl' python j/I/jobrBWwTu with job batch system ID: 10 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T09:02:06.270719811Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:02:06.271744145Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpvVBxV8/worker_log.txt STDERR: 2019-10-11T09:02:06.290250183Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:02:06.293056924Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmphpsbhB/worker_log.txt STDERR: 2019-10-11T09:02:11.447692443Z INFO:toil.leader:Job ended successfully: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/validate.cwl' python G/0/jobzBCSc4 STDERR: 2019-10-11T09:02:11.477289143Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python 9/8/jobMrBtrM with job batch system ID: 11 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T09:02:11.478363996Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_email.cwl' python f/S/jobW0HJQ7 with job batch system ID: 12 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T09:02:11.507584144Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/upload_to_synapse.cwl' python j/I/jobrBWwTu STDERR: 2019-10-11T09:02:11.509113570Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO with job batch system ID: 13 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T09:02:12.337178115Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:02:12.344123990Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpLQF_O_/worker_log.txt STDERR: 2019-10-11T09:02:12.348998809Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:02:12.350455384Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmptDNiEy/worker_log.txt STDERR: 2019-10-11T09:02:12.358912475Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:02:12.360240640Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpq3MQCx/worker_log.txt STDERR: 2019-10-11T09:02:16.828447582Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python 9/8/jobMrBtrM STDERR: 2019-10-11T09:02:16.829365606Z INFO:toil.leader:Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/submit_to_challenge.cwl' python D/t/jobBT4ymB with job batch system ID: 14 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T09:02:16.829549317Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO STDERR: 2019-10-11T09:02:16.830026348Z WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO STDERR: 2019-10-11T09:02:16.830163739Z WARNING:toil.leader:O/e/jobgAbzCO INFO:toil.worker:---TOIL WORKER OUTPUT LOG--- STDERR: 2019-10-11T09:02:16.830207919Z WARNING:toil.leader:O/e/jobgAbzCO INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932. STDERR: 2019-10-11T09:02:16.830301040Z WARNING:toil.leader:O/e/jobgAbzCO [job annotate_submission.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh/6/d/out_tmpdiryWBX6M$ python \ STDERR: 2019-10-11T09:02:16.830344569Z WARNING:toil.leader:O/e/jobgAbzCO annotationSubmission.py \ STDERR: 2019-10-11T09:02:16.830435920Z WARNING:toil.leader:O/e/jobgAbzCO -s \ STDERR: 2019-10-11T09:02:16.830476440Z WARNING:toil.leader:O/e/jobgAbzCO 9693830 \ STDERR: 2019-10-11T09:02:16.830547170Z WARNING:toil.leader:O/e/jobgAbzCO -v \ STDERR: 2019-10-11T09:02:16.830617760Z WARNING:toil.leader:O/e/jobgAbzCO /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmprSwxdd/stge96a5b5a-ccd6-4874-a04c-596d3cc904b8/results.json \ STDERR: 2019-10-11T09:02:16.830673861Z WARNING:toil.leader:O/e/jobgAbzCO -p \ STDERR: 2019-10-11T09:02:16.830745362Z WARNING:toil.leader:O/e/jobgAbzCO true \ STDERR: 2019-10-11T09:02:16.830786461Z WARNING:toil.leader:O/e/jobgAbzCO -f \ STDERR: 2019-10-11T09:02:16.830879381Z WARNING:toil.leader:O/e/jobgAbzCO true \ STDERR: 2019-10-11T09:02:16.830920302Z WARNING:toil.leader:O/e/jobgAbzCO -c \ STDERR: 2019-10-11T09:02:16.831010312Z WARNING:toil.leader:O/e/jobgAbzCO /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmprSwxdd/stgde9a4a84-33ab-4fcf-bb37-30329cbf99ce/.synapseConfig STDERR: 2019-10-11T09:02:16.831100572Z WARNING:toil.leader:O/e/jobgAbzCO INFO:cwltool:[job annotate_submission.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh/6/d/out_tmpdiryWBX6M$ python \ STDERR: 2019-10-11T09:02:16.831129622Z WARNING:toil.leader:O/e/jobgAbzCO annotationSubmission.py \ STDERR: 2019-10-11T09:02:16.831175503Z WARNING:toil.leader:O/e/jobgAbzCO -s \ STDERR: 2019-10-11T09:02:16.831216123Z WARNING:toil.leader:O/e/jobgAbzCO 9693830 \ STDERR: 2019-10-11T09:02:16.831317864Z WARNING:toil.leader:O/e/jobgAbzCO -v \ STDERR: 2019-10-11T09:02:16.831435234Z WARNING:toil.leader:O/e/jobgAbzCO /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmprSwxdd/stge96a5b5a-ccd6-4874-a04c-596d3cc904b8/results.json \ STDERR: 2019-10-11T09:02:16.831614864Z WARNING:toil.leader:O/e/jobgAbzCO -p \ STDERR: 2019-10-11T09:02:16.844984853Z WARNING:toil.leader:O/e/jobgAbzCO true \ STDERR: 2019-10-11T09:02:16.845105693Z WARNING:toil.leader:O/e/jobgAbzCO -f \ STDERR: 2019-10-11T09:02:16.845118504Z WARNING:toil.leader:O/e/jobgAbzCO true \ STDERR: 2019-10-11T09:02:16.845124484Z WARNING:toil.leader:O/e/jobgAbzCO -c \ STDERR: 2019-10-11T09:02:16.845275505Z WARNING:toil.leader:O/e/jobgAbzCO /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmprSwxdd/stgde9a4a84-33ab-4fcf-bb37-30329cbf99ce/.synapseConfig STDERR: 2019-10-11T09:02:16.845394325Z WARNING:toil.leader:O/e/jobgAbzCO Welcome, ehrdreamservice! STDERR: 2019-10-11T09:02:16.845421235Z WARNING:toil.leader:O/e/jobgAbzCO STDERR: 2019-10-11T09:02:16.845575766Z WARNING:toil.leader:O/e/jobgAbzCO Traceback (most recent call last): STDERR: 2019-10-11T09:02:16.845639045Z WARNING:toil.leader:O/e/jobgAbzCO File "annotationSubmission.py", line 75, in STDERR: 2019-10-11T09:02:16.845808286Z WARNING:toil.leader:O/e/jobgAbzCO _with_retry(lambda: annotate_submission(syn, args.submissionid, args.annotation_values, to_public=args.to_public, force_change_annotation_acl=args.force_change_annotation_acl),wait=3,retries=10) STDERR: 2019-10-11T09:02:16.845894567Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/retry.py", line 105, in _with_retry STDERR: 2019-10-11T09:02:16.845998278Z WARNING:toil.leader:O/e/jobgAbzCO six.reraise(exc_info[0], exc_info[1], exc_info[2]) STDERR: 2019-10-11T09:02:16.846190547Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/retry.py", line 53, in _with_retry STDERR: 2019-10-11T09:02:16.846350139Z WARNING:toil.leader:O/e/jobgAbzCO response = function() STDERR: 2019-10-11T09:02:16.846442638Z WARNING:toil.leader:O/e/jobgAbzCO File "annotationSubmission.py", line 75, in STDERR: 2019-10-11T09:02:16.846616679Z WARNING:toil.leader:O/e/jobgAbzCO _with_retry(lambda: annotate_submission(syn, args.submissionid, args.annotation_values, to_public=args.to_public, force_change_annotation_acl=args.force_change_annotation_acl),wait=3,retries=10) STDERR: 2019-10-11T09:02:16.846798920Z WARNING:toil.leader:O/e/jobgAbzCO File "annotationSubmission.py", line 61, in annotate_submission STDERR: 2019-10-11T09:02:16.846976460Z WARNING:toil.leader:O/e/jobgAbzCO status = syn.store(status) STDERR: 2019-10-11T09:02:16.847180662Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/client.py", line 912, in store STDERR: 2019-10-11T09:02:16.847238912Z WARNING:toil.leader:O/e/jobgAbzCO return type(obj)(**self.restPUT(obj.putURI(), obj.json())) STDERR: 2019-10-11T09:02:16.847326752Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/client.py", line 3368, in restPUT STDERR: 2019-10-11T09:02:16.847408072Z WARNING:toil.leader:O/e/jobgAbzCO exceptions._raise_for_status(response, verbose=self.debug) STDERR: 2019-10-11T09:02:16.847547082Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/synapseclient/exceptions.py", line 153, in _raise_for_status STDERR: 2019-10-11T09:02:16.847606193Z WARNING:toil.leader:O/e/jobgAbzCO raise SynapseHTTPError(message, response=response) STDERR: 2019-10-11T09:02:16.847712163Z WARNING:toil.leader:O/e/jobgAbzCO synapseclient.exceptions.SynapseHTTPError: 412 Client Error: STDERR: 2019-10-11T09:02:16.847793554Z WARNING:toil.leader:O/e/jobgAbzCO Submission Status: 9693830 was updated since you last fetched it, retrieve it again and reapply the update STDERR: 2019-10-11T09:02:16.847895994Z WARNING:toil.leader:O/e/jobgAbzCO [job annotate_submission.cwl] Max memory used: 45MiB STDERR: 2019-10-11T09:02:16.847978974Z WARNING:toil.leader:O/e/jobgAbzCO INFO:cwltool:[job annotate_submission.cwl] Max memory used: 45MiB STDERR: 2019-10-11T09:02:16.848032944Z WARNING:toil.leader:O/e/jobgAbzCO [job annotate_submission.cwl] completed permanentFail STDERR: 2019-10-11T09:02:16.848116075Z WARNING:toil.leader:O/e/jobgAbzCO WARNING:cwltool:[job annotate_submission.cwl] completed permanentFail STDERR: 2019-10-11T09:02:16.848181275Z WARNING:toil.leader:O/e/jobgAbzCO Traceback (most recent call last): STDERR: 2019-10-11T09:02:16.848264375Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/toil/worker.py", line 331, in workerScript STDERR: 2019-10-11T09:02:16.848323155Z WARNING:toil.leader:O/e/jobgAbzCO job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) STDERR: 2019-10-11T09:02:16.848424796Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1378, in _runner STDERR: 2019-10-11T09:02:16.848521136Z WARNING:toil.leader:O/e/jobgAbzCO returnValues = self._run(jobGraph, fileStore) STDERR: 2019-10-11T09:02:16.848617087Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/toil/job.py", line 1323, in _run STDERR: 2019-10-11T09:02:16.848681536Z WARNING:toil.leader:O/e/jobgAbzCO return self.run(fileStore) STDERR: 2019-10-11T09:02:16.848783537Z WARNING:toil.leader:O/e/jobgAbzCO File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 606, in run STDERR: 2019-10-11T09:02:16.848835347Z WARNING:toil.leader:O/e/jobgAbzCO raise cwltool.errors.WorkflowException(status) STDERR: 2019-10-11T09:02:16.848921158Z WARNING:toil.leader:O/e/jobgAbzCO WorkflowException: permanentFail STDERR: 2019-10-11T09:02:16.848977148Z WARNING:toil.leader:O/e/jobgAbzCO ERROR:toil.worker:Exiting the worker because of a failed job on host fb0a4e18f2b6 STDERR: 2019-10-11T09:02:16.849137638Z WARNING:toil.leader:O/e/jobgAbzCO WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO with ID O/e/jobgAbzCO to 0 STDERR: 2019-10-11T09:02:16.849974372Z WARNING:toil.leader:Job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO with ID O/e/jobgAbzCO is completely failed STDERR: 2019-10-11T09:02:17.536308186Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_email.cwl' python f/S/jobW0HJQ7 STDERR: 2019-10-11T09:02:17.649865633Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:02:17.651042818Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpsUaR_v/worker_log.txt STDERR: 2019-10-11T09:02:21.260331349Z INFO:toil.leader:Job ended successfully: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/submit_to_challenge.cwl' python D/t/jobBT4ymB STDERR: 2019-10-11T09:02:21.260517391Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python 9/8/jobMrBtrM with job batch system ID: 15 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T09:02:21.947581207Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:02:21.948908681Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpg46MYY/worker_log.txt STDERR: 2019-10-11T09:02:22.039889007Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python 9/8/jobMrBtrM STDERR: 2019-10-11T09:02:22.042349566Z INFO:toil.leader:Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/validate.cwl' python G/0/jobzBCSc4 with job batch system ID: 16 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T09:02:22.666455150Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:02:22.667485404Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmpHfx4e1/worker_log.txt STDERR: 2019-10-11T09:02:22.770922494Z INFO:toil.leader:Job ended successfully: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/validate.cwl' python G/0/jobzBCSc4 STDERR: 2019-10-11T09:02:22.773802695Z INFO:toil.leader:Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python p/7/job9mPjVw with job batch system ID: 17 and cores: 1, disk: 2.0 G, and memory: 100.0 M STDERR: 2019-10-11T09:02:23.389965881Z DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/tmpGCnCgh'. STDERR: 2019-10-11T09:02:23.391183415Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/toil-b92e907b-ab81-4f7e-b952-e90c2e822b0b-e89ed93837262f1d6b20cd9e97e84170/tmp9Ei3aR/worker_log.txt STDERR: 2019-10-11T09:02:23.499023062Z INFO:toil.leader:Job ended successfully: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/download_from_synapse.cwl' python p/7/job9mPjVw STDERR: 2019-10-11T09:02:33.595430779Z INFO:toil.leader:Finished toil run with 8 failed jobs. STDERR: 2019-10-11T09:02:33.596009471Z INFO:toil.leader:Failed jobs at end of the run: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_training_docker.cwl' python C/y/job4RYdld 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_submission_docker.cwl' python R/t/jobUZXriU 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/get_docker_config.cwl' python m/p/jobcphbSj 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/upload_to_synapse.cwl' python j/I/jobrBWwTu 'CWLWorkflow' 8/U/jobniYcvf 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/f53b45dd-556d-46a7-bac6-935095ca0d04/EHR-challenge-develop/run_synthetic_infer_docker.cwl' python E/S/jobt6lgEA 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/validate_docker.cwl' python C/h/job3b7SVR 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v1.6/annotate_submission.cwl' python O/e/jobgAbzCO STDERR: 2019-10-11T09:02:33.683849084Z Traceback (most recent call last): STDERR: 2019-10-11T09:02:33.683877244Z File "/usr/local/bin/toil-cwl-runner", line 10, in STDERR: 2019-10-11T09:02:33.683882544Z sys.exit(main()) STDERR: 2019-10-11T09:02:33.683886614Z File "/usr/local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 1276, in main STDERR: 2019-10-11T09:02:33.684134945Z outobj = toil.start(wf1) STDERR: 2019-10-11T09:02:33.684176775Z File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 781, in start STDERR: 2019-10-11T09:02:33.684298716Z return self._runMainLoop(rootJobGraph) STDERR: 2019-10-11T09:02:33.684308816Z File "/usr/local/lib/python2.7/site-packages/toil/common.py", line 1054, in _runMainLoop STDERR: 2019-10-11T09:02:33.684594787Z jobCache=self._jobCache).run() STDERR: 2019-10-11T09:02:33.684607187Z File "/usr/local/lib/python2.7/site-packages/toil/leader.py", line 246, in run STDERR: 2019-10-11T09:02:33.684611617Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore) STDERR: 2019-10-11T09:02:33.684632247Z toil.leader.FailedJobsException

Created by Ivan Brugere ivanbrugere
Hi @trberg , I noticed that this model was re-launched this morning (not by me). I suspected you or someone else were debugging this issue. I re-ran on my own to confirm. Now this runs to completion on the "Challenge Submission" channel. Was there a change on your end? Thanks, Ivan
Hi @ivanbrugere, The Error on our end was: Traceback (most recent call last): File "/app/train.py", line 12, in data_train, labels_train, labels_train_expand = model_includes.read_ehrdc_data(config["train path"]) File "/app/includes/models.py", line 35, in read_ehrdc_data data[k] = pd.read_csv(f, usecols=useful_keys[k]) File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 685, in parser_f return _read(filepath_or_buffer, kwds) File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 457, in _read parser = TextFileReader(fp_or_buf, **kwds) File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 895, in __init__ self._make_engine(self.engine) File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 1135, in _make_engine self._engine = CParserWrapper(self.f, **self.options) File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 1959, in __init__ _validate_usecols_names(usecols, self.orig_names) File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 1255, in _validate_usecols_names "columns expected but not found: {missing}".format(missing=missing) ValueError: Usecols do not match columns, columns expected but not found: ['time_of_birth'] This looks like an oversight on my part. I'll make the adjustment to the synthetic data to remove the 'time_of_birth' column. In the meantime, you can go ahead and ignore that column as it's not in our data. Thank you, Tim

"EHR DREAM Challenge Submission" channel failed page is loading…