Fusing 2D Material World Knowledge on 3D Geometry
trailer.3.mp4
🌐 Project Page 📄 Paper (Arxiv) 📦 Materials ✍🏻 Citation
We propose MatSpray, a framework for fusing 2D material world knowledge from diffusion models into 3D geometry to obtain relightable assets with spatially varying physically based rendering (PBR) materials. Our method leverages pretrained 2D diffusion-based material predictors to generate per-view material maps (base color, roughness, metallic) and integrates them into a 3D Gaussian Splatting representation via Gaussian ray tracing. To enhance multi-view consistency and physical accuracy, we introduce a lightweight Neural Merger that refines material estimates using a softmax-based restriction. Our approach enables high-quality relightable 3D reconstruction that outperforms existing methods in both quantitative metrics and visual realism, while being approximately 3.5× faster than state-of-the-art inverse rendering approaches.
Paper is currently under review, we will release the code shortly in a cleaned version, star the repo to not miss the release. Up until then if you have any questions regarding the project or need material to compare against fast, feel free to contact us.
You can find our paper on arXiv, please consider citing, if you find this work useful:
@inproceedings{langsteiner2025matspray,
author = {Langsteiner, Philipp and Dihlmann, Jan-Niklas and Lensch, Hendrik P.A.},
title = {MatSpray: Fusing 2D Material World Knowledge on 3D Geometry},
booktitle = {arXiv preprint},
year = {2025}
}