PBR-SR: Mesh PBR Texture Super Resolution from 2D Image Priors

Abstract

We present PBR-SR, a novel method for physically based rendering (PBR) texture super resolution (SR). It outputs high-resolution, high-quality PBR textures from low-resolution (LR) PBR input in a zero-shot manner. PBR-SR leverages an off-the-shelf super-resolution model trained on natural images, and iteratively minimizes the deviations between super-resolution priors and differentiable renderings. These enhancements are then back-projected into the PBR map space in a differentiable manner to produce refined, high-resolution textures. To mitigate view inconsistencies and lighting sensitivity, which is common in view-based super-resolution, our method applies 2D prior constraints across multi-view renderings, iteratively refining the shared, upscaled textures. In parallel, we incorporate identity constraints directly in the PBR texture domain to ensure the upscaled textures remain faithful to the LR input. PBR-SR operates without any additional training or data requirements, relying entirely on pretrained image priors. We demonstrate that our approach produces high-fidelity PBR textures for both artist-designed and AI-generated meshes, outperforming both direct SR models application and prior texture optimization methods. Our results show high-quality outputs in both PBR and rendering evaluations, supporting advanced applications such as relighting.

BibTeX

@inproceedings{chenpbr,
  title={PBR-SR: Mesh PBR Texture Super Resolution from 2D Image Priors},
  author={Chen, Yujin and Nie, Yinyu and Ummenhofer, Benjamin and Birkl, Reiner and Paulitsch, Michael and Nie{\ss}ner, Matthias},
  booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems}
}

License

PBR-SR is released under the MIT License.

This codebase is built on top of Paint-it, which is also MIT-licensed. The combined LICENSE file preserves both copyright notices.

If you use PBR-SR in your research, please cite both works — see CITATION.cff.

---

Installation

Step 1 — Clone the repository

git clone https://github.com/TerenceCYJ/PBR-SR.git
cd PBR-SR
git submodule update --init --recursive

Step 2 — Make CUDA 11.8 + gcc 10 visible

PBR-SR uses nvdiffrast and other CUDA extensions that JIT-compile on first import. The system toolchain must be on PATH before creating the environment or launching the pipeline:

• CUDA toolkit 11.8 (must match the PyTorch CUDA wheel — see requirements.txt)
• gcc 10.x (CUDA 11.8 does not support gcc ≥ 12)
• ninja build tool
• NVIDIA GPU with compute capability ≥ 7.0

A typical setup:

export CUDA_HOME=/usr/local/cuda-11.8
export PATH=$CUDA_HOME/bin:/usr/bin/gcc-10:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
export CC=gcc-10
export CXX=g++-10

Step 3 — Create the Conda environment

conda env create -f environment.yml
conda activate pbr_sr

Alternatively, using pip:

pip install -r requirements.txt
pip install ninja   # if not already pulled in

Step 4 — Verify the setup

which ninja  && ninja --version          # → 1.x
which nvcc   && nvcc --version | tail -1 # → CUDA 11.8
which gcc    && gcc --version | head -1  # → gcc 10.x
python tests/smoke_imports.py            # → "OK: all 8 modules imported cleanly."
python tests/check_release_files.py      # → verifies repo-local configs are present

If any of these fail, fix them before continuing — and before downloading weights.

Step 5 — Download pre-trained weights

Place the following checkpoints under weights/ before running:

weights/
├── DiffBIR_v2.1.pt                       # DiffBIR v2.1 IRControlNet
├── sd2.1-base-zsnr-laionaes5.ckpt        # Stable Diffusion 2.1 base used by DiffBIR
├── realesrgan_s4_swinir_100k.pth         # SwinIR + Real-ESRGAN stage-1 cleaner for DiffBIR
├── CAMixerSR/
│   ├── CAMixerSRx2_DF.pth
│   ├── CAMixerSRx4_DF.pth
│   └── ...
└── HiT_SR/
    ├── HiT-SRF-2x.pth
    └── HiT-SRF-4x.pth

See weights/README.md for download URLs. The DiffBIR loader will auto-download the three loose files from HuggingFace on first run if they are missing, but pre-placing them avoids the 6 GB download.

Note: Always run from the repository root — model loaders resolve weights/ and third_party/ relative to the current working directory.

---

Usage

Scale-4 optimization on the bundled Cradle_Globe example:

python run_pbr_sr.py \
    --data_dir './data/LR_mesh/Cgtrader/Cradle_Globe' \
    --out_dir  './outputs/Cradle_Globe_scale4' \
    --n_iter 10000 \
    --kd_init_method 'DiffBIR_x4' \
    --ks_init_method 'CAMixer_HiT_SR_x4' \
    --n_init_method  'US_bicubic' \
    --pbr_sr_scale 4 \
    --img_sr_model 'DiffBIR_x4' \
    --render_cache_dir  './outputs/cache_render' \
    --diffbir_steps 10 \
    --robust_opti

Note: The default argument values already match the bundled Cradle_Globe example, so all path flags above can be omitted when reproducing the default scene.

⚠️ First-run cache generation (~30-40 minutes) — On the very first launch the script renders 750 camera views and runs DiffBIR SR on each to build an offline pseudo-GT cache. The script appears stalled during this phase but is healthy. Monitor progress with:

watch -n 5 'ls outputs/cache_render/MESH_Cradle_Globe_*/ | wc -l'

Subsequent runs reuse the cache and skip this phase entirely.

Note on --n_iter — The iteration count is rounded up to the next full epoch. With the default camera settings, one epoch is 188 batches; --n_iter 10000 runs ~10 040 steps.

---

SR Backends

Bundled (work out of the box)

Included as Git submodules under third_party/:

Backend	Flag	Notes
DiffBIR	--img_sr_model DiffBIR_x4	Default; used by the README example
CAMixerSR	--ks_init_method CAMixer_HiT_SR_x4	Texture init
HiT_SR	--ks_init_method CAMixer_HiT_SR_x4	Transformer SR backbone
SwinIR	--img_sr_model SwinIR_*	Alternative backbone
StableSR	--img_sr_model StableSR_x4	Alternative pipeline

Optional (install separately)

Clone into third_party/ and place weights under weights/ to enable:

Backend	Flag	Upstream repo
OSEDiff	--img_sr_model OSEDiff_SR_x{2,4}	https://github.com/cswry/OSEDiff
InvSR	--img_sr_model InvSR_x4	https://github.com/zsyOAOA/InvSR
HAT	--img_sr_model HAT_SR_x{2,4}	https://github.com/XPixelGroup/HAT

---

Project layout

PBR-SR/
├── run_pbr_sr.py                 # main CLI entry point
├── pbr_sr/                       # PBR-SR Python package
│   ├── data_pipeline.py
│   ├── paths.py                  # central path / env-var config
│   ├── utils.py, utils_online.py
│   ├── sr_utils.py
│   └── sr_backends/              # DiffBIR / OSEDiff / StableSR / SwinIR wrappers
├── configs/                      # PBR-SR-owned backend configs (not submodule files)
├── nvdiff_render/                # differentiable PBR rasterizer
├── data/                         # bundled Cradle_Globe mesh + envmaps
├── weights/                      # pre-trained checkpoints (gitignored)
├── third_party/                  # SR backend submodules
├── tests/smoke_imports.py
├── docs/                         # additional documentation
└── legacy/                       # historical code (do not import)

For advanced configuration (custom paths, cluster setups) see docs/env_vars.md.