Current methods let to reconstruct 3D interiors with high-quality geometry and texture. However, they struggle with the environments with mirrors and glass.
A recent paper suggests a method for identifying mirrors and estimating mirror surface depth on RGBD data collected with commodity hardware.
Mirror regions are identified based on color information, and the mirror is modeled as a plane. The mirror’s position in 3D is predicted by using an estimated mirror normal and the information from the mirror’s surroundings. The researchers annotated 3D mirror planes in three popular RGBD datasets and established benchmarks for the mirror plane prediction task. It is shown that the suggested architecture helps to improve mirror depth estimates, significantly mitigating 3D reconstruction artifacts due to mirror surfaces.
Despite recent progress in depth sensing and 3D reconstruction, mirror surfaces are a significant source of errors. To address this problem, we create the Mirror3D dataset: a 3D mirror plane dataset based on three RGBD datasets (Matterport3D, NYUv2 and ScanNet) containing 7,011 mirror instance masks and 3D planes. We then develop Mirror3DNet: a module that refines raw sensor depth or estimated depth to correct errors on mirror surfaces. Our key idea is to estimate the 3D mirror plane based on RGB input and surrounding depth context, and use this estimate to directly regress mirror surface depth. Our experiments show that Mirror3DNet significantly mitigates errors from a variety of input depth data, including raw sensor depth and depth estimation or completion methods.
Research paper: Tan, J., Lin, W., Chang, A. X., and Savva, M., “Mirror3D: Depth Refinement for Mirror Surfaces”, 2021. Link: https://arxiv.org/abs/2106.06629