Dear Participants,
Thank you for your active participation in the Testing Phase of the challenge. Please carefully review the following instructions regarding Docker submissions:
After you submit your Docker image, our system will automatically execute it.
1. **Received** indicates that your submission has been successfully uploaded.
2. **Accepted** indicates that your Docker container has executed successfully and produced valid outputs. You will also receive an email notification when your submission is accepted.
3. **Invalid** indicates that your Docker container did not run successfully or produced invalid outputs. In this case, you will receive an email notification. Please check the submission logs, make the necessary modifications, and resubmit.
**Submission Limits:**
1. The number of Accepted submissions is limited to **two (2)** per team per task. And for final ranking we will use your last ACCEPTED submission before the deadline.
2. There is no limit on submissions that result in other statuses (e.g., Invalid).
**System Constraint:**
1. Docker containers are executed with a memory limit of **24 GB**. Please ensure your image can run within this constraint.
2. ⚠️ All containers are run in a completely **offline** environment. Do not attempt to download any files or packages inside your Docker container; such attempts will fail and the submission will be marked Invalid.
**Output Requirement:**
1. Your container must write the prediction results as .nii.gz files inside the **/output** directory.
2. ⚠️ Do not compress results into .zip files.
**Scoring and Ranking:**
1. Scores will not be displayed in the Synapse tables during the Testing Phase.
2. All submissions will be evaluated internally, and teams will be ranked accordingly.
3. Top three teams for each task will be contacted in **mid-September**.
4. The full official rankings will be published on **September 27**.
We appreciate your efforts and contributions, and we look forward to your continued participation.
Best regards,
The LISA 2025 Challenge Organizers
Created by LISA Challenge LISA_mri_challenge Hi @Mohamad-Bashar-Dasoki
The organisers mentioned that only the latest accepted submission version will automatically be counted and you don't have to do anything in this case.
Hope this helps! Hi, i push new container with the same tag (latest) after submission, and I want to count the last one
help please Dear @vcasellesb
Thank you very much for your response and explanation. I came up with an idea to deal with that problem, which is to pass one piece of data at a time and clear the system memory. That way, if we pass them one by one, the out of memory problem does not seem to occur. Dear @Rafther0112,
For dealing with your message, I found useful running my inference code with as little processes as possible (I'm assuming you use multiprocessing).
However, that wasn't enough. The error message by torch disappeared, but my container was failing constantly due to out of RAM issues. I had to get rid of multiprocessing altogether (no parallelization whatsoever, run the pipeline sequentially, like @sharat.s.e kindly suggested).
This is how I managed to get a submission to run correctly. Please let me known if you have further questions.
Best,
Vicent Dear @vcasellesb
I hope you are well. I would like to know how you managed to deal with the "No space left on device (28)" problem. I am having the same problem and have not been able to solve it yet. Hi @NazishKhalid123,
Please check your log. It seems you are using the cpu which causes the issue.
Best regards,
The LISA 2025 Challenge Organizers Hi @LISA_mri_challenge,
I hope this message finds you well. I submitted my Docker image for Task 2 [Hippocampus/Basal Ganglia Segmentation – specify which one] at approximately 6:23 AM (US Central Time) today. The submission ID is 31609c43-6f8d-4b8….
The status has remained in “Evaluating” since then. I also noticed that the evaluation files show updated timestamps (e.g., last modified at 1:16 PM), but the job has not completed or produced results.
Could you please check whether this is due to a system backlog or if my submission has become stuck? I request you to please look into this matter looking farward to hear from you.
Thank you very much for your support.
Hi @sharat.s.e,
The number of Accepted submissions is limited to two (2) per team per task. And for final ranking we will use your last ACCEPTED submission before the deadline.
Best,
The LISA 2025 Challenge Organizers Dear @LISA_mri_challenge,
Thank you for posting the instructions for the testing phase and clarifying things.
I had a query regarding the number of submissions we can make.
You mentioned that the number of submissions is limited to two.
1. Is this per task?
2. If there are 2 docker submissions per task, will both be evaluated separately and the best of two considered for final scoring?
Regards,
Sharat Dear @vcasellesb,
You are most welcome.
What I meant is that I am running multiple processes sequentially one after the other, which will require only one worker.
If you have the optimal model from nnUNet, perhaps you could try to manually build the inference pipeline by loading the trained model weights.
From what I read, when you run nnUNetv2_plan_and_preprocess, it evaluates your system configuration and optimizes the pipeline accordingly. So, perhaps the container is trying to build an inference environment as per your setup and causing the submission issues. By the way, I am not using the nnUNet implementation for my final submission.
Regarding point 2, it is recommended to free the gpu memory after each batch is processed.
Hope you are able to resolve the issues and submit in time. Good luck to you.
Regards,
Sharat Dear @Musti,
Thanks for replying, I appreciate it!
Best,
Vicent Hi @vcasellesb,
No I have not faced this problem and don't actually recognize it. Hope you manage to solve it soon, and good luck with the submission!
Best,
Musti Dear @sharat.s.e,
Thanks so much for your input, it is greatly appreciated.
I thought running containers in challenges with --ipc=host was the norm, I guess I was mistaken...
First, what do you mean by "sequential multiprocessing on GPU with single worker"? Doesn't this mean NO multiprocessing? If you process sequentially with 1 single worker?
I am trying now what you mentioned, iterating for all files and processing each one separately. Before I was running HD-BET for obtaining a brain bounding box - which uses nnUNet - on all the testing cases first. nnUNet uses multiprocessing for preprocessing and segmentation export (converting from logits to seg classes, etc). Honestly, this is as far as the algorithm can go without breaking. It breaks when trying to segment the second case with HD-BET... I guess maybe it's the multiprocessing stuff, which adds too much overhead for the server to handle?
About your second message:
1. Yes, I haven't even tried anything other than BS==1 haha
2. This might be interesting... I am doing that AFTER ALL cases have been processed, not after each one has.
3. I haven't tried this but I don't know if I have time hahaha
Anyways, thank you so much for your help. You have definitely pointed me towards many solutions.
Best,
Vicent
@vcasellesb, not sure if it's relevant to the error you are observing, but to avoid GPU OOM error and limit memory usage, you can consider the following if not already done.
1. Reduce the inference batch size to 1. So it loads only one image to the memory at a time and processes it.
2. Encapsulate the cuda operations in a function and use torch.cuda.empty_cache() and gc.collect() to free memory after the scope of function execution.
3. Move some processing operations to cpu, which might take up more processing time but will help to reduce the gpu memory requirement.
Hope this helps. Hi @vcasellesb,
I have not run into this issue thus far. I am using sequential multiprocessing on GPU with single worker in my inference algorithm.
Do you really require multiple workers (and perhaps parallel processing) by setting host ipc for the inference workflow?
I am not sure if the container evaluation script will include this flag for spawning the docker.
Regards,
Sharat Hi,
I wanted to give an update. I've reduced the amount of processes the code uses to 1, and also tweaked some parameters to reduce memory consumption. It still fails.
Just for a comparison, running the same container with 6 processes allowed and without said tweaks on my 20 GB RAM server with a 12GB gpu runs perfectly. The only caveat is that you have to supply the command as follows:
```
docker container run --rm --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -m 14G ...
```
These flags are from the [Nvidia official documentation](https://docs.nvidia.com/deeplearning/frameworks/user-guide/index.html#setincshmem), section about running models on containers.
Anyways, please let me know if there is something that can be done wrt this issue. Otherwise, I won't be able to submit.
Best,
Vicent I'm sorry because I've been submitting a lot of jobs/containers, I am gradually reducing the number of processes my algorithm uses to see if this solves the issue. Anyways, I think it would be common to run the jobs with the shared memory of the host (i.e., by setting --ipc=host). Could this be done?
How have you guys managed this, @Musti, @sharat.s.e? Did you run into this issue? Do you use multiprocessing within your algorithm?
Vicent Hi @LISA_mri_challenge,
I am getting the following error message:
```
RuntimeError: unable to write to file : No space left on device (28)
```
From what I've [read online,](https://discuss.pytorch.org/t/device-out-of-space-error-when-torch-save-with-multiprocessing-in-docker/167869/2) it seems to be a problem with the container not being run with enough shared memory. Is the docker container run with the "--ipc==host" flag?
Best,
Vicent Hi @NazishKhalid123,
Yes, we run on a GPU environment.
Best,
The LISA 2025 Challenge Organizers Will our docker files will run in the GPU environment? As I assume that it is been runned in GPU environment.