Efficient 3D Surface Super-resolution via Normal-based Multimodal Restoration

High-fidelity 3D surface is essential for vision tasks across various domains such as medical imaging, cultural heritage preservation, quality inspection, virtual reality, and autonomous navigation. However, the intricate nature of 3D data representations poses significant challenges in restoring diverse 3D surfaces while capturing fine-grained geometric details at a low cost. This paper introduces an efficient multimodal normal-based 3D surface super-resolution (mn3DSSR) framework, designed to address the challenges of microgeometry enhancement and computational overhead. Specifically, we have constructed one of the largest photometric stereo dataset, ensuring superior data quality and diversity through meticulous subjective selection. Furthermore, we explore a new two-branch multimodal alignment approach along with a multimodal split fusion module to mitigate computational complexity while improving restoration performance. To address the limitations associated with normal-based multimodal learning, we develop novel normal-induced loss functions that facilitate geometric consistency and improve feature alignment. Extensive experiments conducted on seven benchmark datasets across four different 3D data representations demonstrate that mn3DSSR consistently outperforms state-of-the-art super-resolution methods in terms of restoration accuracy with high computational efficiency.

A. Multimodal Pre-processing Stage (MPS)
MPS performs necessary feature transformation on the raw multimodal data, addressing two main issues: (1) normalizing the input to minimize variations and distribution differences across modalities from various sources, and (2) constructing primary features for the alignment module, generating shape and texture features.

B. Multimodal Swin-Transformer Alignment (MSTA)
To align the previously processed I′_lr and D′_lr, we propose a new two-branch multimodal Swin-Transformer alignment (MSTA) module, consisting of a RGB-texture alignment branch and a depth-shape alignment branch. Specifically, our objective is primarily to enhance the texture normal information by aligning I′_lr with N^t_lr in the RGB-texture branch. Simultaneously, we inject the global 3D geometric information into the shape normal by aligning D′_lr with N^s_lr in the depth-shape branch.

C. Multimodal Split Fusion (MSF)
After processing the side-modality features in the MSTA module, Ftn and Fsn are further fused into the normal modality to assist in super-resolution feature extraction. In our previous method [11], a fusion module has been developed based on a spatial feature transform to modulate side-modalities. However, this approach often neglects dynamic changes in the main normal branch and struggles to adapt to the distinct characteristics of texture and shape normal features. To address this limitation, we propose a multimodal split fusion (MSF) module that integrates texture and shape normal features.

D. Proposed normal-based dataset

Several normal-based datasets have been established for 3D surface processing. Nonetheless, we continue to face the following fundamental challenges in training our mn3DSSR model: (i) a notable scarcity of diverse 3D surface shape datasets, particularly within open-source repositories; (ii) significant difficulties in obtaining high-quality normal maps and corresponding multimodal data that represent fine-grained surface details and complex geometries; and (iii) the limited scale of existing high-quality normal datasets, which is typically insufficient in size to support the training of robust deep super-resolution models.

To address these challenges, we propose to establish a dedicated and large-scale normal-based multimodal dataset acquired using photometric stereo setups. The essential motivation lies in the fact that photometric stereo is typically more accurate for computing surface normals than other methods based on single image prediction, such as shape-from-shading or deep learning models. Fig. 1 shows typical samples from our dataset. The normal modality N_gt provides fine-grained geometry information. Meanwhile, the depth modality D_gt provides a continuous 3D surface constraint. In contrast, the RGB modality I_gt contains 18 images captured under diverse calibrated lightings, which represents complex texture and material features that provide rich visual cues for multimodal surface processing.

Compared to our previous wonderful photometric stereo (WPS) dataset, we summarize six main improvements in WPS+ as follows.

1) We have re-captured low-quality samples from WPS to improve the overall data quality.

2) WPS captures RGB images through the camera sensor without Gamma correction, leading to inaccurate normal acquisition. In contrast, WPS+ has addressed this issue by applying Gamma correction before obtaining surface normal maps.

3) To synthesize the best normal maps, we have adopted three different photometric stereo-based methods [39], [40], and [41], which have enhanced the overall quality of 3D surface reconstruction. In contrast, WPS relies on one least square based Lambertian method [42].

4) We have invited three professionals to carefully evaluate and select the best 3D reconstruction results, investing over 1,000 hours to ensure high-quality samples.

5) To better represent diverse surface shapes, we have greatly expanded the number of dataset samples, now including 600 objects in WPS+ compared to 400 objects in WPS.

6) After improving data quality, we have scaled the WPS+ dataset by a larger magnification to obtain a more comprehensive super-resolution dataset, including the ×2, ×4, and ×8 sampling settings.