Hi, We noticed that the per-frame tool masks loaded in metrics.py are overwritten before metrics are computed: ``` for i in range(len(masks)): masks[i] = global_mask_t # replaces tool masks with overlap mask ``` As a result, PSNR/SSIM/LPIPS are evaluated over the full reprojection overlap region, including tool pixels. This contradicts the FAQ: "Will tools be excluded from the final evaluation metrics? Yes. We provide tool masks and load them in the metric scripts." Could you confirm the intended behavior and release a corrected metrics.py if needed? Thanks!

Created by Sergio Canar sergiocanar
Hi all, I pushed the fix to the repo, but if you don't want to re-pull, just make these two changes in metrics.py: In readImages (~line 203): ``` frame_mask = frame_mask * overlap_mask # was: frame_mask = overlap_mask ``` In evaluate (~line 250): ``` masks[i] = masks[i] * global_mask_t # was: masks[i] = global_mask_t ``` Metrics are now computed only over pixels that are both tool free and inside the overlap region, as intended. Thanks again for catching this!
Thanks for pointing this out! That is not the intended behavior. I will fix the metrics script and push to the github again. Thanks again :)

Question about tool mask handling in metrics.py Task 2 page is loading…