", line 3, in raise_from STDERR: 2023-02-13T22:40:53.324150518Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request STDERR: 2023-02-13T22:40:53.324153616Z httplib_response = conn.getresponse() STDERR: 2023-02-13T22:40:53.324156674Z File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse STDERR: 2023-02-13T22:40:53.324159847Z response.begin() STDERR: 2023-02-13T22:40:53.324162698Z File "/usr/local/lib/python3.8/http/client.py", line 316, in begin STDERR: 2023-02-13T22:40:53.324165850Z version, status, reason = self._read_status() STDERR: 2023-02-13T22:40:53.324168657Z File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status STDERR: 2023-02-13T22:40:53.324171886Z line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") STDERR: 2023-02-13T22:40:53.324175087Z File "/usr/local/lib/python3.8/socket.py", line 669, in readinto STDERR: 2023-02-13T22:40:53.324178158Z return self._sock.recv_into(b) STDERR: 2023-02-13T22:40:53.324180937Z socket.timeout: timed out STDERR: 2023-02-13T22:40:53.324183722Z ``` Could you please check this error? Thank you in advance. " />

Hi, I confirmed our docker image ran successfully and I initially got prediction probability depending on participants. I submitted the model and received confirmation and submission accepted email. However, I also received workflow failed email which shows errors in urllib3 (scoket.timeout: timed out) - which I don't understand since inference.py doesn't make any outside request? Error message is : ``` DERR: 2023-02-13T22:40:53.324138006Z Traceback (most recent call last): STDERR: 2023-02-13T22:40:53.324141006Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request STDERR: 2023-02-13T22:40:53.324144389Z six.raise_from(e, None) STDERR: 2023-02-13T22:40:53.324147329Z File "", line 3, in raise_from STDERR: 2023-02-13T22:40:53.324150518Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request STDERR: 2023-02-13T22:40:53.324153616Z httplib_response = conn.getresponse() STDERR: 2023-02-13T22:40:53.324156674Z File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse STDERR: 2023-02-13T22:40:53.324159847Z response.begin() STDERR: 2023-02-13T22:40:53.324162698Z File "/usr/local/lib/python3.8/http/client.py", line 316, in begin STDERR: 2023-02-13T22:40:53.324165850Z version, status, reason = self._read_status() STDERR: 2023-02-13T22:40:53.324168657Z File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status STDERR: 2023-02-13T22:40:53.324171886Z line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") STDERR: 2023-02-13T22:40:53.324175087Z File "/usr/local/lib/python3.8/socket.py", line 669, in readinto STDERR: 2023-02-13T22:40:53.324178158Z return self._sock.recv_into(b) STDERR: 2023-02-13T22:40:53.324180937Z socket.timeout: timed out STDERR: 2023-02-13T22:40:53.324183722Z ``` Could you please check this error? Thank you in advance.

Created by Jouhyun Jeon Clare
@Clare @Sakb Your submissions are successful and accepted now. Please continue with your writeup submission.
@vjbytes102 I ran the docker image on the randomly selected 36 audio files (from a training set) and there was no problem. Also, running the image using the full dataset worked perfectly. Plus, current model that I submitted today is little bit different from the previous model that I submitted during the leaderboard so I think there's no specific reason to receive an error message on my end... I wondering if you can help me to diagnose the error message? It would be great if you can run the container in non-detached mode so that we can get better output on what the error is. Since I am not importing urllib, I am wondering the reason to receive urllib-related timeout error message.
@Clare we did not detect any error on our end, to make sure i re-run your code with correct memory configurations but still got the failure. It appears to me that the issue is on your end with the docker image
Thank you for your help @vjbytes102 However, I received a same error. I received both submission accepted email and workflow fail email two times. It is weird. I found out that max memory is different between 1st and 2nd log files. my error message (2nd attempt) is: ``` STDERR: 2023-02-13T23:27:33.593140332Z 915a75a9bc13 2023-02-13 23:27:33,592 MainThread INFO cwltool: Resolved '/var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/workflow.cwl' to 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/workflow.cwl' STDERR: 2023-02-13T23:27:45.647881726Z 915a75a9bc13 2023-02-13 23:27:45,647 MainThread WARNING cwltool: Workflow checker warning: STDERR: 2023-02-13T23:27:45.647937467Z TB-DREAM-Challenge-Infra-main/workflow.cwl:72:9: Source 'docker_repository' of type ["null", STDERR: 2023-02-13T23:27:45.647944522Z "string"] may be incompatible STDERR: 2023-02-13T23:27:45.647948276Z TB-DREAM-Challenge-Infra-main/workflow.cwl:172:9: with sink 'docker_repository' of type "string" STDERR: 2023-02-13T23:27:45.647951530Z TB-DREAM-Challenge-Infra-main/workflow.cwl:73:9: Source 'docker_digest' of type ["null", "string"] STDERR: 2023-02-13T23:27:45.647955152Z may be incompatible STDERR: 2023-02-13T23:27:45.647958277Z TB-DREAM-Challenge-Infra-main/workflow.cwl:174:9: with sink 'docker_digest' of type "string" STDERR: 2023-02-13T23:27:48.000667087Z 915a75a9bc13 2023-02-13 23:27:48,000 MainThread INFO toil: Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe. STDERR: 2023-02-13T23:27:50.104790570Z 915a75a9bc13 2023-02-13 23:27:50,104 MainThread INFO toil.leader: 1 jobs are running, 0 jobs are issued and waiting to run STDERR: 2023-02-13T23:27:51.505723627Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp2ppqbogh/worker_log.txt STDERR: 2023-02-13T23:27:53.264660239Z 915a75a9bc13 2023-02-13 23:27:53,264 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-3ysg8m4a with job batch system ID: 1 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-13T23:27:53.264861293Z 915a75a9bc13 2023-02-13 23:27:53,264 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-fvph_4q0 with job batch system ID: 2 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-13T23:27:53.265049855Z 915a75a9bc13 2023-02-13 23:27:53,264 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-brp_4lt9 with job batch system ID: 3 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-13T23:27:53.265234946Z 915a75a9bc13 2023-02-13 23:27:53,265 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-8nq7cvzn with job batch system ID: 4 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-13T23:27:53.265418815Z 915a75a9bc13 2023-02-13 23:27:53,265 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-4yf3z61l with job batch system ID: 5 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-13T23:27:58.484758700Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp1ud8wy2h/worker_log.txt STDERR: 2023-02-13T23:27:58.910826648Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpt7mfzwcb/worker_log.txt STDERR: 2023-02-13T23:28:06.969228691Z 915a75a9bc13 2023-02-13 23:28:06,968 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-fvph_4q0 STDERR: 2023-02-13T23:28:07.239510004Z 915a75a9bc13 2023-02-13 23:28:07,239 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/set_permissions.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_set_permissions.cwl/instance-3ysg8m4a STDERR: 2023-02-13T23:28:12.689017618Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpyr31ewre/worker_log.txt STDERR: 2023-02-13T23:28:12.944952148Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp7c5gyese/worker_log.txt STDERR: 2023-02-13T23:28:21.435802250Z 915a75a9bc13 2023-02-13 23:28:21,432 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-4yf3z61l STDERR: 2023-02-13T23:28:24.220466974Z 915a75a9bc13 2023-02-13 23:28:24,220 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-brp_4lt9 STDERR: 2023-02-13T23:28:24.229296796Z 915a75a9bc13 2023-02-13 23:28:24,229 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-zdzm9_em with job batch system ID: 6 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-13T23:28:29.879042652Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpddcd1hq7/worker_log.txt STDERR: 2023-02-13T23:28:31.784624194Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp7fky1eda/worker_log.txt STDERR: 2023-02-13T23:28:33.413694162Z 915a75a9bc13 2023-02-13 23:28:33,413 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-zdzm9_em STDERR: 2023-02-13T23:28:33.414736228Z 915a75a9bc13 2023-02-13 23:28:33,414 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-4nhqr6kr with job batch system ID: 7 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-13T23:28:36.803842509Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp7bdid1ab/worker_log.txt STDERR: 2023-02-13T23:28:39.736862657Z 915a75a9bc13 2023-02-13 23:28:39,736 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-4nhqr6kr STDERR: 2023-02-13T23:28:39.738108664Z 915a75a9bc13 2023-02-13 23:28:39,737 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-iuutc881 with job batch system ID: 8 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-13T23:28:41.243442548Z 915a75a9bc13 2023-02-13 23:28:41,230 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-8nq7cvzn STDERR: 2023-02-13T23:28:41.243481043Z 915a75a9bc13 2023-02-13 23:28:41,231 MainThread INFO toil.leader: Issued job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys with job batch system ID: 9 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-13T23:28:41.243486335Z 915a75a9bc13 2023-02-13 23:28:41,231 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-o8fewj_8 with job batch system ID: 10 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-13T23:28:41.243501846Z 915a75a9bc13 2023-02-13 23:28:41,231 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-elfgq929 with job batch system ID: 11 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-13T23:28:47.746881392Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp92labq61/worker_log.txt STDERR: 2023-02-13T23:28:49.027313044Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/worker_log.txt STDERR: 2023-02-13T23:28:55.224779277Z 915a75a9bc13 2023-02-13 23:28:55,224 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-iuutc881 STDERR: 2023-02-13T23:28:59.754063219Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpmhsryigz/worker_log.txt STDERR: 2023-02-13T23:29:10.390312437Z 915a75a9bc13 2023-02-13 23:29:10,389 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-elfgq929 STDERR: 2023-02-13T23:29:13.217146098Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpgjhszgr5/worker_log.txt STDERR: 2023-02-13T23:29:21.892246475Z 915a75a9bc13 2023-02-13 23:29:21,891 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-o8fewj_8 STDERR: 2023-02-13T23:29:21.893103228Z 915a75a9bc13 2023-02-13 23:29:21,892 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-u9ppmdb8 with job batch system ID: 12 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-13T23:29:26.275481164Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpf2e38hok/worker_log.txt STDERR: 2023-02-13T23:29:27.778853667Z 915a75a9bc13 2023-02-13 23:29:27,778 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/check_status.cwl' kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_check_status.cwl/instance-u9ppmdb8 STDERR: 2023-02-13T23:29:27.779638816Z 915a75a9bc13 2023-02-13 23:29:27,779 MainThread INFO toil.leader: Issued job 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-o8fewj_8 with job batch system ID: 13 and cores: 1, disk: 1.0 G, and memory: 100.0 M STDERR: 2023-02-13T23:29:30.495458436Z INFO:toil.worker:Redirecting logging to /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmpver2jaik/worker_log.txt STDERR: 2023-02-13T23:29:31.111741723Z 915a75a9bc13 2023-02-13 23:29:31,111 MainThread INFO toil.leader: Job ended: 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_email.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_email.cwl/instance-o8fewj_8 STDERR: 2023-02-13T23:40:46.470141401Z 915a75a9bc13 2023-02-13 23:40:46,468 MainThread INFO toil.leader: Job ended: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys STDERR: 2023-02-13T23:40:46.471211077Z 915a75a9bc13 2023-02-13 23:40:46,470 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys STDERR: 2023-02-13T23:40:46.471219851Z 915a75a9bc13 2023-02-13 23:40:46,470 MainThread WARNING toil.leader: Log from job kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys follows: STDERR: 2023-02-13T23:40:46.471237092Z =========> STDERR: 2023-02-13T23:40:46.471240740Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG--- STDERR: 2023-02-13T23:40:46.471244182Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe. STDERR: 2023-02-13T23:40:46.471247156Z [job run_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z$ python3 \ STDERR: 2023-02-13T23:40:46.471250595Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py \ STDERR: 2023-02-13T23:40:46.471254024Z -s \ STDERR: 2023-02-13T23:40:46.471256894Z 9732155 \ STDERR: 2023-02-13T23:40:46.471259732Z -p \ STDERR: 2023-02-13T23:40:46.471262460Z docker.synapse.org/syn45594136/docker-final-submission \ STDERR: 2023-02-13T23:40:46.471274654Z -d \ STDERR: 2023-02-13T23:40:46.471277584Z sha256:15d4fcd9edf148399d7833466c14b3f3524c9adba1cf79a3247e8b495704c9d4 \ STDERR: 2023-02-13T23:40:46.471280489Z --store \ STDERR: 2023-02-13T23:40:46.471283221Z --status \ STDERR: 2023-02-13T23:40:46.471286701Z VALIDATED \ STDERR: 2023-02-13T23:40:46.471289350Z --parentid \ STDERR: 2023-02-13T23:40:46.471291997Z syn51089588 \ STDERR: 2023-02-13T23:40:46.471294508Z -c \ STDERR: 2023-02-13T23:40:46.471297375Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/tmpgti9xhmz/stg51ae2487-cc26-4cf3-8dfa-9fb18e5c5e36/.synapseConfig \ STDERR: 2023-02-13T23:40:46.471313161Z -t \ STDERR: 2023-02-13T23:40:46.471315819Z 2 STDERR: 2023-02-13T23:40:46.471319447Z INFO:cwltool:[job run_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z$ python3 \ STDERR: 2023-02-13T23:40:46.471322983Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py \ STDERR: 2023-02-13T23:40:46.471326341Z -s \ STDERR: 2023-02-13T23:40:46.471329067Z 9732155 \ STDERR: 2023-02-13T23:40:46.471332045Z -p \ STDERR: 2023-02-13T23:40:46.471334656Z docker.synapse.org/syn45594136/docker-final-submission \ STDERR: 2023-02-13T23:40:46.471337670Z -d \ STDERR: 2023-02-13T23:40:46.471340246Z sha256:15d4fcd9edf148399d7833466c14b3f3524c9adba1cf79a3247e8b495704c9d4 \ STDERR: 2023-02-13T23:40:46.471343121Z --store \ STDERR: 2023-02-13T23:40:46.471345667Z --status \ STDERR: 2023-02-13T23:40:46.471348217Z VALIDATED \ STDERR: 2023-02-13T23:40:46.471350624Z --parentid \ STDERR: 2023-02-13T23:40:46.471353617Z syn51089588 \ STDERR: 2023-02-13T23:40:46.471356373Z -c \ STDERR: 2023-02-13T23:40:46.471358777Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/tmpgti9xhmz/stg51ae2487-cc26-4cf3-8dfa-9fb18e5c5e36/.synapseConfig \ STDERR: 2023-02-13T23:40:46.471361972Z -t \ STDERR: 2023-02-13T23:40:46.471364417Z 2 STDERR: 2023-02-13T23:40:46.471366852Z Welcome, Vijay Yadav! STDERR: 2023-02-13T23:40:46.471370057Z STDERR: 2023-02-13T23:40:46.471372613Z STDERR: 2023-02-13T23:40:46.471387291Z UPGRADE AVAILABLE STDERR: 2023-02-13T23:40:46.471390211Z STDERR: 2023-02-13T23:40:46.471392965Z A more recent version of the Synapse Client (2.7.0) is available. Your version (2.5.1) can be upgraded by typing: STDERR: 2023-02-13T23:40:46.471399956Z pip install --upgrade synapseclient STDERR: 2023-02-13T23:40:46.471402825Z STDERR: 2023-02-13T23:40:46.471405559Z Python Synapse Client version 2.7.0 release notes STDERR: 2023-02-13T23:40:46.471408017Z STDERR: 2023-02-13T23:40:46.471410622Z https://python-docs.synapse.org/build/html/news.html STDERR: 2023-02-13T23:40:46.471413377Z STDERR: 2023-02-13T23:40:46.471416044Z root STDERR: 2023-02-13T23:40:46.471418744Z mounting volumes STDERR: 2023-02-13T23:40:46.471421507Z checking for containers STDERR: 2023-02-13T23:40:46.471424128Z running container STDERR: 2023-02-13T23:40:46.471426659Z creating logfile STDERR: 2023-02-13T23:40:46.471429197Z Traceback (most recent call last): STDERR: 2023-02-13T23:40:46.471431783Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request STDERR: 2023-02-13T23:40:46.471434623Z six.raise_from(e, None) STDERR: 2023-02-13T23:40:46.471437233Z File "", line 3, in raise_from STDERR: 2023-02-13T23:40:46.471440165Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request STDERR: 2023-02-13T23:40:46.471443265Z httplib_response = conn.getresponse() STDERR: 2023-02-13T23:40:46.471446225Z File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse STDERR: 2023-02-13T23:40:46.471449481Z response.begin() STDERR: 2023-02-13T23:40:46.471464566Z File "/usr/local/lib/python3.8/http/client.py", line 316, in begin STDERR: 2023-02-13T23:40:46.471467637Z version, status, reason = self._read_status() STDERR: 2023-02-13T23:40:46.471470365Z File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status STDERR: 2023-02-13T23:40:46.471475809Z line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") STDERR: 2023-02-13T23:40:46.471478618Z File "/usr/local/lib/python3.8/socket.py", line 669, in readinto STDERR: 2023-02-13T23:40:46.471481987Z return self._sock.recv_into(b) STDERR: 2023-02-13T23:40:46.471484753Z socket.timeout: timed out STDERR: 2023-02-13T23:40:46.471487343Z STDERR: 2023-02-13T23:40:46.471490243Z During handling of the above exception, another exception occurred: STDERR: 2023-02-13T23:40:46.471493321Z STDERR: 2023-02-13T23:40:46.471495813Z Traceback (most recent call last): STDERR: 2023-02-13T23:40:46.471499110Z File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 440, in send STDERR: 2023-02-13T23:40:46.471502181Z resp = conn.urlopen( STDERR: 2023-02-13T23:40:46.471505195Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen STDERR: 2023-02-13T23:40:46.471507959Z retries = retries.increment( STDERR: 2023-02-13T23:40:46.471510895Z File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment STDERR: 2023-02-13T23:40:46.471517884Z raise six.reraise(type(error), error, _stacktrace) STDERR: 2023-02-13T23:40:46.471520724Z File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise STDERR: 2023-02-13T23:40:46.471523784Z raise value STDERR: 2023-02-13T23:40:46.471526609Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen STDERR: 2023-02-13T23:40:46.471541769Z httplib_response = self._make_request( STDERR: 2023-02-13T23:40:46.471544933Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 451, in _make_request STDERR: 2023-02-13T23:40:46.471547757Z self._raise_timeout(err=e, url=url, timeout_value=read_timeout) STDERR: 2023-02-13T23:40:46.471550660Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout STDERR: 2023-02-13T23:40:46.471553541Z raise ReadTimeoutError( STDERR: 2023-02-13T23:40:46.471556124Z urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) STDERR: 2023-02-13T23:40:46.471559003Z STDERR: 2023-02-13T23:40:46.471561781Z During handling of the above exception, another exception occurred: STDERR: 2023-02-13T23:40:46.471564713Z STDERR: 2023-02-13T23:40:46.471567250Z Traceback (most recent call last): STDERR: 2023-02-13T23:40:46.471569981Z File "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py", line 252, in STDERR: 2023-02-13T23:40:46.471573485Z main(syn, args) STDERR: 2023-02-13T23:40:46.471576361Z File "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py", line 193, in main STDERR: 2023-02-13T23:40:46.471579921Z while container in client.containers.list(ignore_removed=True): STDERR: 2023-02-13T23:40:46.471583991Z File "/usr/local/lib/python3.8/site-packages/docker/models/containers.py", line 952, in list STDERR: 2023-02-13T23:40:46.471587070Z resp = self.client.api.containers(all=all, before=before, STDERR: 2023-02-13T23:40:46.471590261Z File "/usr/local/lib/python3.8/site-packages/docker/api/container.py", line 209, in containers STDERR: 2023-02-13T23:40:46.471593185Z res = self._result(self._get(u, params=params), True) STDERR: 2023-02-13T23:40:46.471596302Z File "/usr/local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner STDERR: 2023-02-13T23:40:46.471599837Z return f(self, *args, **kwargs) STDERR: 2023-02-13T23:40:46.471602522Z File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get STDERR: 2023-02-13T23:40:46.471620317Z return self.get(url, **self._set_request_timeout(kwargs)) STDERR: 2023-02-13T23:40:46.471623558Z File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 542, in get STDERR: 2023-02-13T23:40:46.471626564Z return self.request('GET', url, **kwargs) STDERR: 2023-02-13T23:40:46.471629117Z File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 529, in request STDERR: 2023-02-13T23:40:46.471632230Z resp = self.send(prep, **send_kwargs) STDERR: 2023-02-13T23:40:46.471635193Z File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 645, in send STDERR: 2023-02-13T23:40:46.471637978Z r = adapter.send(request, **kwargs) STDERR: 2023-02-13T23:40:46.471640801Z File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 532, in send STDERR: 2023-02-13T23:40:46.471643793Z raise ReadTimeout(e, request=request) STDERR: 2023-02-13T23:40:46.471646557Z requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) STDERR: 2023-02-13T23:40:46.471649668Z [job run_docker.cwl] Max memory used: 19MiB STDERR: 2023-02-13T23:40:46.471652147Z INFO:cwltool:[job run_docker.cwl] Max memory used: 19MiB STDERR: 2023-02-13T23:40:46.471654805Z [job run_docker.cwl] Job error: STDERR: 2023-02-13T23:40:46.471657533Z ("Error collecting output for parameter 'predictions':\nTB-DREAM-Challenge-Infra-main/steps/run_docker.cwl:65:7: Did not find output file with glob pattern: '['predictions.csv']'", {}) STDERR: 2023-02-13T23:40:46.471660590Z ERROR:cwltool:[job run_docker.cwl] Job error: STDERR: 2023-02-13T23:40:46.471663346Z ("Error collecting output for parameter 'predictions':\nTB-DREAM-Challenge-Infra-main/steps/run_docker.cwl:65:7: Did not find output file with glob pattern: '['predictions.csv']'", {}) STDERR: 2023-02-13T23:40:46.471667044Z [job run_docker.cwl] completed permanentFail STDERR: 2023-02-13T23:40:46.471669769Z WARNING:cwltool:[job run_docker.cwl] completed permanentFail STDERR: 2023-02-13T23:40:46.471672443Z Traceback (most recent call last): STDERR: 2023-02-13T23:40:46.471675409Z File "/usr/local/lib/python3.8/site-packages/toil/worker.py", line 366, in workerScript STDERR: 2023-02-13T23:40:46.471679339Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) STDERR: 2023-02-13T23:40:46.471682542Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1392, in _runner STDERR: 2023-02-13T23:40:46.471698377Z returnValues = self._run(jobGraph, fileStore) STDERR: 2023-02-13T23:40:46.471701584Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1329, in _run STDERR: 2023-02-13T23:40:46.471704700Z return self.run(fileStore) STDERR: 2023-02-13T23:40:46.471707653Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 937, in run STDERR: 2023-02-13T23:40:46.471710440Z raise cwltool.errors.WorkflowException(status) STDERR: 2023-02-13T23:40:46.471713277Z cwltool.errors.WorkflowException: permanentFail STDERR: 2023-02-13T23:40:46.471719190Z ERROR:toil.worker:Exiting the worker because of a failed job on host 915a75a9bc13 STDERR: 2023-02-13T23:40:46.471722452Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys to 0 STDERR: 2023-02-13T23:40:46.471726407Z <========= STDERR: 2023-02-13T23:40:46.489520979Z 915a75a9bc13 2023-02-13 23:40:46,489 MainThread WARNING toil.leader: Job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys is completely failed STDERR: 2023-02-13T23:40:59.984032393Z 915a75a9bc13 2023-02-13 23:40:59,983 MainThread INFO toil.leader: Finished toil run with 9 failed jobs. STDERR: 2023-02-13T23:40:59.984348979Z 915a75a9bc13 2023-02-13 23:40:59,984 MainThread INFO toil.leader: Failed jobs at end of the run: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-4yf3z61l 'CWLWorkflow' kind-CWLWorkflow/instance-fd9vo75k 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-zdzm9_em 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-brp_4lt9 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-4nhqr6kr 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-elfgq929 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-iuutc881 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-8nq7cvzn STDERR: 2023-02-13T23:41:00.040493963Z Traceback (most recent call last): STDERR: 2023-02-13T23:41:00.040523148Z File "/usr/local/bin/toil-cwl-runner", line 8, in STDERR: 2023-02-13T23:41:00.040527842Z sys.exit(main()) STDERR: 2023-02-13T23:41:00.040531118Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 1691, in main STDERR: 2023-02-13T23:41:00.040534543Z outobj = toil.start(wf1) STDERR: 2023-02-13T23:41:00.040538513Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 829, in start STDERR: 2023-02-13T23:41:00.040542046Z return self._runMainLoop(rootJobGraph) STDERR: 2023-02-13T23:41:00.040545161Z File "/usr/local/lib/python3.8/site-packages/toil/common.py", line 1115, in _runMainLoop STDERR: 2023-02-13T23:41:00.040548646Z return Leader(config=self.config, STDERR: 2023-02-13T23:41:00.040551955Z File "/usr/local/lib/python3.8/site-packages/toil/leader.py", line 269, in run STDERR: 2023-02-13T23:41:00.040555242Z raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore) STDERR: 2023-02-13T23:41:00.040755837Z toil.leader.FailedJobsException: The job store 'file:/var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/tmp9z6ejoye' contains 9 failed jobs: 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_docker_config.cwl' python3 kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_docker_config.cwl/instance-4yf3z61l, 'CWLWorkflow' kind-CWLWorkflow/instance-fd9vo75k, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/determine_question.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_determine_question.cwl/instance-zdzm9_em, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/get_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_get_submission.cwl/instance-brp_4lt9, 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/get_goldstandard_id.cwl' kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_get_goldstandard_id.cwl/instance-4nhqr6kr, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/annotate_submission.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_annotate_submission.cwl/instance-elfgq929, 'https://raw.githubusercontent.com/Sage-Bionetworks-Workflows/cwl-tool-synapseclient/v1.4/cwl/synapse-get-tool.cwl#synapse-get' synapse kind-https_raw.githubusercontent.com_Sage-Bionetworks-Workflows_cwl-tool-synapseclient_v1.4_cwl_synapse-get-tool.cwl_synapse-get/instance-iuutc881, 'https://raw.githubusercontent.com/Sage-Bionetworks/ChallengeWorkflowTemplates/v3.1/cwl/validate_docker.cwl' challengeutils kind-https_raw.githubusercontent.com_Sage-Bionetworks_ChallengeWorkflowTemplates_v3.1_cwl_validate_docker.cwl/instance-8nq7cvzn STDERR: 2023-02-13T23:41:00.040782150Z Log from job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys follows: STDERR: 2023-02-13T23:41:00.040787110Z =========> STDERR: 2023-02-13T23:41:00.040790540Z INFO:toil.worker:---TOIL WORKER OUTPUT LOG--- STDERR: 2023-02-13T23:41:00.040793579Z INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe. STDERR: 2023-02-13T23:41:00.040796734Z [job run_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z$ python3 \ STDERR: 2023-02-13T23:41:00.040800412Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py \ STDERR: 2023-02-13T23:41:00.040804743Z -s \ STDERR: 2023-02-13T23:41:00.040807648Z 9732155 \ STDERR: 2023-02-13T23:41:00.040811047Z -p \ STDERR: 2023-02-13T23:41:00.040814054Z docker.synapse.org/syn45594136/docker-final-submission \ STDERR: 2023-02-13T23:41:00.040817177Z -d \ STDERR: 2023-02-13T23:41:00.040820027Z sha256:15d4fcd9edf148399d7833466c14b3f3524c9adba1cf79a3247e8b495704c9d4 \ STDERR: 2023-02-13T23:41:00.040823076Z --store \ STDERR: 2023-02-13T23:41:00.040825861Z --status \ STDERR: 2023-02-13T23:41:00.040828500Z VALIDATED \ STDERR: 2023-02-13T23:41:00.040831431Z --parentid \ STDERR: 2023-02-13T23:41:00.040834271Z syn51089588 \ STDERR: 2023-02-13T23:41:00.040837244Z -c \ STDERR: 2023-02-13T23:41:00.040840503Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/tmpgti9xhmz/stg51ae2487-cc26-4cf3-8dfa-9fb18e5c5e36/.synapseConfig \ STDERR: 2023-02-13T23:41:00.040844139Z -t \ STDERR: 2023-02-13T23:41:00.040847063Z 2 STDERR: 2023-02-13T23:41:00.040904755Z INFO:cwltool:[job run_docker.cwl] /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z$ python3 \ STDERR: 2023-02-13T23:41:00.040910145Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py \ STDERR: 2023-02-13T23:41:00.040914005Z -s \ STDERR: 2023-02-13T23:41:00.040922531Z 9732155 \ STDERR: 2023-02-13T23:41:00.040925870Z -p \ STDERR: 2023-02-13T23:41:00.040938033Z docker.synapse.org/syn45594136/docker-final-submission \ STDERR: 2023-02-13T23:41:00.040941120Z -d \ STDERR: 2023-02-13T23:41:00.040944164Z sha256:15d4fcd9edf148399d7833466c14b3f3524c9adba1cf79a3247e8b495704c9d4 \ STDERR: 2023-02-13T23:41:00.040947533Z --store \ STDERR: 2023-02-13T23:41:00.040950281Z --status \ STDERR: 2023-02-13T23:41:00.040952948Z VALIDATED \ STDERR: 2023-02-13T23:41:00.040955880Z --parentid \ STDERR: 2023-02-13T23:41:00.040958965Z syn51089588 \ STDERR: 2023-02-13T23:41:00.040961607Z -c \ STDERR: 2023-02-13T23:41:00.040964715Z /var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/tmpgti9xhmz/stg51ae2487-cc26-4cf3-8dfa-9fb18e5c5e36/.synapseConfig \ STDERR: 2023-02-13T23:41:00.040968033Z -t \ STDERR: 2023-02-13T23:41:00.040970868Z 2 STDERR: 2023-02-13T23:41:00.040973619Z Welcome, Vijay Yadav! STDERR: 2023-02-13T23:41:00.040976564Z STDERR: 2023-02-13T23:41:00.040979352Z STDERR: 2023-02-13T23:41:00.040982128Z UPGRADE AVAILABLE STDERR: 2023-02-13T23:41:00.040985022Z STDERR: 2023-02-13T23:41:00.040988062Z A more recent version of the Synapse Client (2.7.0) is available. Your version (2.5.1) can be upgraded by typing: STDERR: 2023-02-13T23:41:00.040991300Z pip install --upgrade synapseclient STDERR: 2023-02-13T23:41:00.040994415Z STDERR: 2023-02-13T23:41:00.040997310Z Python Synapse Client version 2.7.0 release notes STDERR: 2023-02-13T23:41:00.041000824Z STDERR: 2023-02-13T23:41:00.041003796Z https://python-docs.synapse.org/build/html/news.html STDERR: 2023-02-13T23:41:00.041006951Z STDERR: 2023-02-13T23:41:00.041009854Z root STDERR: 2023-02-13T23:41:00.041012850Z mounting volumes STDERR: 2023-02-13T23:41:00.041016266Z checking for containers STDERR: 2023-02-13T23:41:00.041019579Z running container STDERR: 2023-02-13T23:41:00.041022947Z creating logfile STDERR: 2023-02-13T23:41:00.041026030Z Traceback (most recent call last): STDERR: 2023-02-13T23:41:00.041029170Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request STDERR: 2023-02-13T23:41:00.041032249Z six.raise_from(e, None) STDERR: 2023-02-13T23:41:00.041035223Z File "", line 3, in raise_from STDERR: 2023-02-13T23:41:00.041038705Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request STDERR: 2023-02-13T23:41:00.041041913Z httplib_response = conn.getresponse() STDERR: 2023-02-13T23:41:00.041045147Z File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse STDERR: 2023-02-13T23:41:00.041051633Z response.begin() STDERR: 2023-02-13T23:41:00.041054763Z File "/usr/local/lib/python3.8/http/client.py", line 316, in begin STDERR: 2023-02-13T23:41:00.041057897Z version, status, reason = self._read_status() STDERR: 2023-02-13T23:41:00.041060866Z File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status STDERR: 2023-02-13T23:41:00.041064894Z line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") STDERR: 2023-02-13T23:41:00.041068178Z File "/usr/local/lib/python3.8/socket.py", line 669, in readinto STDERR: 2023-02-13T23:41:00.041071421Z return self._sock.recv_into(b) STDERR: 2023-02-13T23:41:00.041074442Z socket.timeout: timed out STDERR: 2023-02-13T23:41:00.041077411Z STDERR: 2023-02-13T23:41:00.041080339Z During handling of the above exception, another exception occurred: STDERR: 2023-02-13T23:41:00.041083531Z STDERR: 2023-02-13T23:41:00.041086463Z Traceback (most recent call last): STDERR: 2023-02-13T23:41:00.041089471Z File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 440, in send STDERR: 2023-02-13T23:41:00.041092716Z resp = conn.urlopen( STDERR: 2023-02-13T23:41:00.041095721Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen STDERR: 2023-02-13T23:41:00.041099095Z retries = retries.increment( STDERR: 2023-02-13T23:41:00.041102216Z File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment STDERR: 2023-02-13T23:41:00.041105722Z raise six.reraise(type(error), error, _stacktrace) STDERR: 2023-02-13T23:41:00.041108843Z File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise STDERR: 2023-02-13T23:41:00.041112007Z raise value STDERR: 2023-02-13T23:41:00.041114795Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen STDERR: 2023-02-13T23:41:00.041131187Z httplib_response = self._make_request( STDERR: 2023-02-13T23:41:00.041134167Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 451, in _make_request STDERR: 2023-02-13T23:41:00.041137866Z self._raise_timeout(err=e, url=url, timeout_value=read_timeout) STDERR: 2023-02-13T23:41:00.041141051Z File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout STDERR: 2023-02-13T23:41:00.041144552Z raise ReadTimeoutError( STDERR: 2023-02-13T23:41:00.041147647Z urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) STDERR: 2023-02-13T23:41:00.041151235Z STDERR: 2023-02-13T23:41:00.041154353Z During handling of the above exception, another exception occurred: STDERR: 2023-02-13T23:41:00.041157762Z STDERR: 2023-02-13T23:41:00.041163817Z Traceback (most recent call last): STDERR: 2023-02-13T23:41:00.041167087Z File "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py", line 252, in STDERR: 2023-02-13T23:41:00.041171095Z main(syn, args) STDERR: 2023-02-13T23:41:00.041213788Z File "/var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/node-177225ce-58b5-439e-aaf8-3ffc78557aca-1cc402dd0e11d5ae18db04a6de87223d/tmp920cbk6x/416b9083-71dd-46ca-bc84-fdb753ae52c4/t7ci81cf_/tmp-out4atjt00z/run_docker.py", line 193, in main STDERR: 2023-02-13T23:41:00.041219232Z while container in client.containers.list(ignore_removed=True): STDERR: 2023-02-13T23:41:00.041222350Z File "/usr/local/lib/python3.8/site-packages/docker/models/containers.py", line 952, in list STDERR: 2023-02-13T23:41:00.041225766Z resp = self.client.api.containers(all=all, before=before, STDERR: 2023-02-13T23:41:00.041229049Z File "/usr/local/lib/python3.8/site-packages/docker/api/container.py", line 209, in containers STDERR: 2023-02-13T23:41:00.041232476Z res = self._result(self._get(u, params=params), True) STDERR: 2023-02-13T23:41:00.041235648Z File "/usr/local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner STDERR: 2023-02-13T23:41:00.041238906Z return f(self, *args, **kwargs) STDERR: 2023-02-13T23:41:00.041242056Z File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get STDERR: 2023-02-13T23:41:00.041245574Z return self.get(url, **self._set_request_timeout(kwargs)) STDERR: 2023-02-13T23:41:00.041248917Z File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 542, in get STDERR: 2023-02-13T23:41:00.041261718Z return self.request('GET', url, **kwargs) STDERR: 2023-02-13T23:41:00.041264992Z File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 529, in request STDERR: 2023-02-13T23:41:00.041268120Z resp = self.send(prep, **send_kwargs) STDERR: 2023-02-13T23:41:00.041271034Z File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 645, in send STDERR: 2023-02-13T23:41:00.041278261Z r = adapter.send(request, **kwargs) STDERR: 2023-02-13T23:41:00.041281349Z File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 532, in send STDERR: 2023-02-13T23:41:00.041284548Z raise ReadTimeout(e, request=request) STDERR: 2023-02-13T23:41:00.041287407Z requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) STDERR: 2023-02-13T23:41:00.041290537Z [job run_docker.cwl] Max memory used: 19MiB STDERR: 2023-02-13T23:41:00.041293596Z INFO:cwltool:[job run_docker.cwl] Max memory used: 19MiB STDERR: 2023-02-13T23:41:00.041296671Z [job run_docker.cwl] Job error: STDERR: 2023-02-13T23:41:00.041299833Z ("Error collecting output for parameter 'predictions':\nTB-DREAM-Challenge-Infra-main/steps/run_docker.cwl:65:7: Did not find output file with glob pattern: '['predictions.csv']'", {}) STDERR: 2023-02-13T23:41:00.041307870Z ERROR:cwltool:[job run_docker.cwl] Job error: STDERR: 2023-02-13T23:41:00.041311156Z ("Error collecting output for parameter 'predictions':\nTB-DREAM-Challenge-Infra-main/steps/run_docker.cwl:65:7: Did not find output file with glob pattern: '['predictions.csv']'", {}) STDERR: 2023-02-13T23:41:00.041314764Z [job run_docker.cwl] completed permanentFail STDERR: 2023-02-13T23:41:00.041317769Z WARNING:cwltool:[job run_docker.cwl] completed permanentFail STDERR: 2023-02-13T23:41:00.041321637Z Traceback (most recent call last): STDERR: 2023-02-13T23:41:00.041324884Z File "/usr/local/lib/python3.8/site-packages/toil/worker.py", line 366, in workerScript STDERR: 2023-02-13T23:41:00.041328307Z job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) STDERR: 2023-02-13T23:41:00.041331526Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1392, in _runner STDERR: 2023-02-13T23:41:00.041335147Z returnValues = self._run(jobGraph, fileStore) STDERR: 2023-02-13T23:41:00.041338337Z File "/usr/local/lib/python3.8/site-packages/toil/job.py", line 1329, in _run STDERR: 2023-02-13T23:41:00.041341524Z return self.run(fileStore) STDERR: 2023-02-13T23:41:00.041344624Z File "/usr/local/lib/python3.8/site-packages/toil/cwl/cwltoil.py", line 937, in run STDERR: 2023-02-13T23:41:00.041347769Z raise cwltool.errors.WorkflowException(status) STDERR: 2023-02-13T23:41:00.041350568Z cwltool.errors.WorkflowException: permanentFail STDERR: 2023-02-13T23:41:00.041353466Z ERROR:toil.worker:Exiting the worker because of a failed job on host 915a75a9bc13 STDERR: 2023-02-13T23:41:00.041356807Z WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///var/lib/docker/volumes/workflow_orchestrator_shared/_data/6e8c1c51-9780-444a-a22c-664eb07b3257/TB-DREAM-Challenge-Infra-main/steps/run_docker.cwl' python3 kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys with ID kind-file_var_lib_docker_volumes_workflow_orchestrator_shared__data_6e8c1c51-9780-444a-a22c-664eb07b3257_TB-DREAM-Challenge-Infra-main_steps_run_docker.cwl/instance-5ehmhlys to 0 STDERR: 2023-02-13T23:41:00.044068912Z <========= ```
I noticed from the log file that the error occurs in `run_docker.py` between the creation of the log file and the finished training section. Plus, according to the code (run_docker.py), log file name should be "projectID__log.txt" however the log file I received was "projectID_logs.txt".
@sieberts Could you please confirm that your backend is okay?
Hello, I have also the same challenge. :(

submission failed page is loading…