Hi, I have been experiencing network hangup or firewall issues when trying to"makeCluster" in R. I know it is not a lack of resources because I check beforehand the number of cores available. ``` library("parallel"); library("foreach"); library("doParallel") nCores = min(16,detectCores() - 1) cl<-makeCluster(nCores,outfile = "runningCh1.txt", type="FORK") ## Stuck here forever. registerDoParallel(cl) ``` Are there any specific library issues with linux that it is having issues simply making clusters? Or are there network or firewall issues? I have been stuck with this issue since round 1 and am feeling frustrated that no one is able to answer this question I had posted so many times before properly. What exact changes should I do to make it work? I need to use "foreach" and "parLapply". Thanks

Created by SA90
We are using the bash shell to run script score_sc#.sh
It might not be enough to replace [[ with [ because [[ is a bit smarter and interprets the conditions differently than [. For one thing, if you have multiple conditions in your statement, they must be in separate brackets with [ whereas [[ lets you combine them like in most other languages. See the post: https://stackoverflow.com/a/3427959/163505 My guess is that you condition isn't doing what it's supposed to, which later causes your eigen issue (e.g. an important file is not being created at the right time in your condition or something.) In principle, bash should be part of your Docker container, but I haven't ran into that issues specifically. Are building you container using ubuntu? I'm wondering if the docker image is being launched somehow with sh as the interpreter, because in Ubuntu /bin/sh points to /bin/**dash** , not /bin/**bash** . E.g., on my desktop: ``` desktopOnBash:~$ dash # Switch to dash desktopOnDash$ if [[ 1 == 1 ]]; then echo True; fi dash: 1: [[: not found desktopOnDash$ bash # Switch back to bash desktopOnBash:~$ if [[ 1 == 1 ]]; then echo True; fi True ``` Which looks like the error you're getting.
Yes, that seemed to be the issue. Trying to use ports outside of the test environment. I followed your and Yuanfang's advise and write a for loop through the different data_test_obs # that would wait for processes to finish. Now, I am facing a new problem. Even though I am specifying to use #!/bin/bash instead of #!/bin/sh to be able to use "[[" within if statements, I am getting the error "[[ not found" meaning that score_sc1.sh is not run using bash score_sc1.sh and cannot use bash syntax. And if I replace [[ with [ it gives me a very weird error message this is not even part of the R code because I put a print statement at the very beginnning. ``` Calls: make.positive.definite -> eigen Execution halted Error in eigen(m, symmetric = TRUE) : infinite or missing values in 'x' Calls: make.positive.definite -> eigen Execution halted ``` submission ID: 9648102 @thomas.yu , @andrewelamb Is the bash script score_sc#.sh not run as bash script score_sc#.sh? If so, could you please link it to use bash? Also, do you have any leads on why I am getting the weird error about eign values? Thanks!
parallel::makeCluster uses socket communication to talk to your workers, so maybe the ports the method uses by default are blocked in the test environment (documentation says it uses by default R_PARALLEL_PORT, then a randomly chosen port in the range 11000:11999). If you want to stay in R, you could try using a method of parallelism that doesn't use network communication (parSapply or mcapply should work), otherwise you can always split the commands in the shell script as described above.
It could be, I'm not familiar enough with those packages to say. If you're getting network/firewall issues it almost has to be.
Thanks @andrewelamb. I am not explicitly trying to access resources outside of the docker, but that may be what is happening with "makeCluster"? Is there a way to check for that? Is there a parameter that I need to use? My submission ID: 9646956 failed because of this. Do you run into any problem trying to run the code snippet in the original post ? Thanks
The problem may be you are trying to access resources outside the docker image. There is no network access allowed inside the Docker image. I'm not sure why you would run into this issue when trying to run processes in parallel, but if you are having network or firewall issues that may be the problem.
Can one of the moderators respond to this and confirm. I would be quite astonished if a standard simple parallelization in R would not work in the remote machines. @thomas.yu @andrewelamb
Hi, SA90,   that library won't work in production environment. you can try to fix till the end of the challenge, but that's hand-blocking a water tap.   you need to first split your file, and parallelize your code outside of R.   Rscript step1.r first section & Rscript step2.r second section && wait Rscript combinestep1and2.r   btw, for your information, i found this challenge actually does not need parallization as I can see right now. maybe you want to think if the method is on the wrong direction?

Network/ Firewall hangup during cluster making page is loading…