我們提出了一種基於學習的方法，用於從單一圖像評估依據視角的環境照明。我們的方法（稱為 LightDistill）學習從可微幾何和紋理分解的框架中提取知識。目標是使用神經網路直接從單一輸入圖像預測環境圖，從而繞過以迭代最佳化求解的需求。我們基於物理的新策略自輸入圖像上取樣像素，並解耦照明顏色與局部光探測的分佈。實驗結果表明，我們提出的方法可以訓練神經網絡，在不到一秒的時間內從單個圖像中有效地導出高質量的環境圖—與耗時的基於優化的其他方法相比有顯著的改進，這些方法通常需要幾分鐘來獲得可比較的結果。

We present a learning-based method for estimating view-dependent environmental lighting from a single image. Our approach (dubbed LightDistill) learns to distill knowledge from a differentiable geometry and texture decomposition framework. The goal is to directly predict the environment map from a single input image using a neural network to bypass the need for solving iterative optimization. Our new physics-based strategy decouples the illumination color and the distribution of a local light probe from a sampled pixel on the input image. The experimental results show that our proposed method can train a neural network to efficiently derive a high-quality environment map from a single image in less than a second—a significant improvement over the timeconsuming optimization-based alternatives that often require a few minutes to obtain comparable results.

List of Tables 3
List of Figures 4
摘 要 6
Abstract 7
1 Introduction 8
2 Related work 10
3 Approach 13
3.1 LightDistill Phase I: Decomposition under varying illuminations . . . . . . 14
3.1.1 Rendering equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.2 Applying nvdiffrec . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.1.3 Loss functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2 LightDistill Phase II: Learning LightDistill . . . . . . . . . . . . . . . . . . . . . . . 16
3.2.1 Training LightDistill MLP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2.2 Distribution of light probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2.3 Stacking light probes from sampled directions . . . . . . . . . . . . . . . . . . . . 18
3.2.4 Loss functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4 Experiments 20
4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.2 Main results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.3 Implementation details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
5 Conclusion 29
A More Details and Results 30
A.1 Deformation of geometry on Globe dataset . . . . . . . . . . . . . . . . . . . . . . . . 30
A.2 Gradient vanishing by nvdiffrec optimization . . . . . . . . . . . . . . . . . . . . . . 30
A.3 Structure of LightDistill MLP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
A.4 More qualitative results on ALP datasets . . . . . . . . . . . . . . . . . . . . . . . . . . 32
A.5 Comparisons of reconstructions on Gold dataset. . . . . . . . . . . . . . . . . . . . 33
A.6 More qualitative results on NeRD datasets . . . . . . . . . . . . . . . . . . . . . . . . 34
Bibliography 40

[1] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, 2022.
[2] M. Boss, R. Braun, V. Jampani, J. T. Barron, C. Liu, and H. P. Lensch. Nerd: Neural reflectance decomposition from image collections. In IEEE International Conference on Computer Vision (ICCV), 2021.
[3] M. Boss, V. Jampani, R. Braun, C. Liu, J. T. Barron, and H. P. Lensch. Neural-pil: Neural pre-integrated lighting for reflectance decomposition. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[4] J. Choi, S. Lee, H. Park, S. Jung, I. Kim, and J. Cho. MAIR: multi-view attention inverse rendering with 3d spatially-varying lighting estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, 2023.
[5] S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 5491–5500. IEEE, 2022.
[6] M. Gardner, Y. Hold-Geoffroy, K. Sunkavalli, C. Gagné, and J. Lalonde. Deep parametric indoor lighting estimation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, 2019.
[7] M. Garon, K. Sunkavalli, S. Hadap, N. Carr, and J. Lalonde. Fast spatially-varying indoor lighting estimation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, 2019.
[8] J. Hasselgren, N. Hofmann, and J. Munkberg. Shape, light, and material decomposition from images using monte carlo rendering and denoising. In NeurIPS, 2022.
[9] Z. Li, M. Shafiei, R. Ramamoorthi, K. Sunkavalli, and M. Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and SVBRDF from a single image. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, 2020.
[10] Z. Li, Z. Xu, R. Ramamoorthi, K. Sunkavalli, and M. Chandraker. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Trans. Graph., 37(6):269, 2018.
[11] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.
Nerf: Representing scenes as neural radiance fields for view synthesis. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, 2020.
[12] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1–102:15, 2022.
[13] J. Munkberg, W. Chen, J. Hasselgren, A. Evans, T. Shen, T. Müller, J. Gao, and S. Fidler. Extracting triangular 3d models, materials, and lighting from images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, 2022.
[14] A. Sommer, U. Schwanecke, and E. Schömer. Real-time light estimation and neural soft shadows for AR indoor scenarios. J. WSCG, 31(1-2):71–79, 2023.
[15] P. P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, 2021.
[16] P. P. Srinivasan, B. Mildenhall, M. Tancik, J. T. Barron, R. Tucker, and N. Snavely. Lighthouse: Predicting lighting volumes for spatially-coherent illumination. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, 2020.
[17] C. Sun, M. Sun, and H. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, 2022.
[18] G. Wang, Y. Yang, C. C. Loy, and Z. Liu. Stylelight: HDR panorama generation for lighting estimation and editing. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, 2022.
[19] H. Yu, S. Agarwala, C. Herrmann, R. Szeliski, N. Snavely, J. Wu, and D. Sun. Accidental light probes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, 2023.
[20] K. Zhang, F. Luan, Q. Wang, K. Bala, and N. Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, 2021.
[21] X. Zhang, P. P. Srinivasan, B. Deng, P. E. Debevec, W. T. Freeman, and J. T. Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. CoRR, abs/2106.01970, 2021.