RenderTarget + WebGPU Backend: 'readRenderTargetPixelsAsync' returns blank data regardless of configuration

[EepyBerry]

Description

I'm unable to read RenderTargets using the WebGPU backend correctly, regardless of which color space is used during creation.
I have tried multiple things, but what I've gathered until now is the following:

• None of the color spaces work when rendering alongside an animation loop: NoColorSpace, LinearSRGBColorSpace, SRGBColorSpace (see RenderTarget + WebGPU backend: Setting color space to SRGBColorSpace in constructor breaks "readRenderTargetPixelsAsync" #31654)
• The only case where I have managed to read the data is when setting the RenderTarget on the renderer before any call to render/renderAsync, then calling the function once... Even then, I cannot reproduce this reliably
• RenderTarget pixels can only be properly read using WebGL, might be due to preserveDrawingBuffer not existing for the WebGPURenderer?
• In the case of WebGL, the scene background color is ignored while rendering to the RenderTarget (see below)

Side note: my code might not be correct, in which case how may I read this data correctly regardless of the backend?

Reproduction steps

(note: check the live example for a complete setup)

1. Create a scene with a basic mesh, using a WebGPURenderer with the WebGPU backend
2. Create a RenderTarget with any color space, and render to it
3. Try to read pixels using readRenderTargetPixelsAsync

Code

import * as THREE from "three";
import {
  LinearSRGBColorSpace,
  MeshStandardNodeMaterial,
  WebGPURenderer,
} from "three/build/three.webgpu";
import { Fn, positionLocal, vec4 } from "three/src/nodes/TSL";

let screenshotted = false;

// IIFE
(async () => {
  const camera = new THREE.PerspectiveCamera(
    70,
    window.innerWidth / window.innerHeight,
    0.1,
    100
  );
  camera.position.z = 2;

  const scene = new THREE.Scene();
  scene.background = new THREE.Color("#111");

  const light = new THREE.PointLight();
  light.position.set(0, 0, 5);
  light.intensity = 100;

  const mesh = new THREE.Mesh(
    new THREE.SphereGeometry(),
    new MeshStandardNodeMaterial({
      colorNode: Fn(() => vec4(positionLocal, 1.0))(),
    })
  );
  scene.add(mesh);
  scene.add(light);

  const renderer = new WebGPURenderer({
    forceWebGL: false, // setting to true restores rendering
  });
  renderer.setPixelRatio(window.devicePixelRatio);
  renderer.setSize(window.innerWidth, window.innerHeight);
  renderer.setAnimationLoop(() => renderer.render(scene, camera));
  document.body.appendChild(renderer.domElement);

  window.addEventListener("keypress", async () => {
    if (screenshotted) return;
    screenshotted = true;
    await screenshot(renderer, scene, camera);
  });
})();

// ----------------------- START test code -----------------------
async function screenshot(
  renderer: WebGPURenderer,
  scene: THREE.Scene,
  camera: THREE.PerspectiveCamera
) {
  const canvas = renderer.domElement;
  const w = canvas.width,
    h = canvas.height;
  const screenshotRTT = new THREE.RenderTarget(w, h, {
    colorSpace: THREE.SRGBColorSpace, // change this to see differences
  });
  renderer.setRenderTarget(screenshotRTT);
  renderer.render(scene, camera);
  renderer.setRenderTarget(null);

  const rawBuffer = new Uint8Array(w * h * 4);
  rawBuffer.set(
    await renderer.readRenderTargetPixelsAsync(screenshotRTT, 0, 0, w, h)
  );
  pixelDataToCanvas(rawBuffer);
  screenshotRTT.dispose();
}

function pixelDataToCanvas(data: Uint8Array) {
  const canvas = document.createElement("canvas");
  canvas.width = window.innerWidth;
  canvas.height = window.innerHeight;
  const ctx = canvas.getContext("2d")!;
  const imgData = ctx.createImageData(window.innerWidth, window.innerHeight);
  imgData.data.set(data);
  ctx.putImageData(imgData, 0, 0);
  document.body.appendChild(canvas);
}
// ------------------------ END test code ------------------------

Live example

• CodeSandbox: https://codesandbox.io/p/sandbox/yhwl56 (note: press a key when focused on the window to trigger the RenderTarget rendering code)

Screenshots

No response

Version

r179

Device

Desktop

Browser

Chrome

OS

Windows

[Mugen87]

I've simplified the code a bit and created a fiddle: https://jsfiddle.net/sr4yqw6h/3/

The code works with WebGL but not with WebGPU. Interestingly, when executing the code in a full screen app, the screenshot routine works as expected. But when in an iframe, I always see:

Required size for texture data layout (3041128) exceeds the linear data size (2959200) with offset (0).

@EepyBerry Do you mind creating a code sandbox with the code from the fiddle? We might have to report a browser issue and we need a test case for reproduction. I can't run the fiddle as a fullscreen app.

[EepyBerry]

Of course, here you go! https://codesandbox.io/p/sandbox/838nqj
Also, I did get the bug you mentioned while running some tests on my side too...

[EepyBerry]

Ah, before I forget, a small thing I've noticed during my tests (originally what I was testing for, thus the reason for the PointLight, etc):
I'm pretty sure that reading pixels from a RenderTarget with WebGPU returns pixels in the correct orientation, as opposed to WebGL where the pixels are flipped in the Y axis.
Is that intended behaviour? And should I make another issue for that?

[Mugen87]

It seems the codesandbox still uses the three version from package.json however we need the current dev version.

Are you able to make a condesandbox without TS and a build, just with plain JS like in the fiddle?

[Mugen87]

I'm pretty sure that reading pixels from a RenderTarget with WebGPU returns pixels in the correct orientation, as opposed to WebGL where the pixels are flipped in the Y axis.

I've noticed that, too. We circumvent this issue by using the readback data as a data texture. Different uv conventions for the back end fix that issue. See https://threejs.org/examples/webgpu_multiple_rendertargets_readback for more details.

I don't know yet why this happens and if it's a bug but it's easy to fix by flipping the buffer data. In WebGL, I believe you don't need a readback for a screenshot at all since you can directly draw renderer.domElement onto a canvas.

[EepyBerry]

It seems the codesandbox still uses the three version from package.json however we need the current dev version.

Are you able to make a condesandbox without TS and a build, just with plain JS like in the fiddle?

Oh, sorry, would this work? https://codesandbox.io/p/sandbox/qlw627
CodeSandbox is really finicky when it comes to importmaps, as it turns out...

[Mugen87]

Oh, sorry, would this work?

Yes, that exactly demonstrates the issue! When performing the readback, it works in fullscreen but not inside the iFrame.

Thanks for the sandbox! I'll file a bug at the Chromium bug tracker the next days.

[Mugen87]

Bug report: https://issues.chromium.org/issues/440538861

[greggman]

I don't think this is a browser issue? unless I'm mis-understanding the issue

copyBufferToTexture and copyTextureToBuffer require bytesPerRow to be a multiple of 256 bytes

That means for example, if you wanted to read a 4x4 rgba8unorm texture (64 bytes). You're required to have a buffer that is (3*256+16) = 784 bytes. The first 3 rows padded to 256 bytes and the last row does not require padding.

WebGL didn't have this issue, you could read a 4x4 texture with gl.readPixels in 64 bytes. But in reality it the underlying graphics APIs had this limit, WebGL was just having to read the data with these alignment limits and then re-pack it (making it slower). WebGPU doesn't do this for you. If you want a packed result then you need to re-pack it yourself or write a compute shader to read out the data packed.

[greggman]

You can see the bug here

three.js/src/renderers/webgpu/utils/WebGPUTextureUtils.js

Line 543 in 0cd49e3

size: width * height * bytesPerTexel,

The code computes bytesPerRow but it doesn't use that in the buffer size calculation.

The buffer size required is (height - 1) * bytesPerRow + width * bytesPerTexel

[Mugen87]

This indeed resolves the WebGPU warning. Thank you for the fix!

So when the user wants to use the data for applying onto a canvas, it's necessary to unpack the data as mentioned in #31658 (comment), right?

If I now apply the readback buffer without unpacking onto the canvas via:

const imgData = ctx.createImageData( canvas.width, canvas.height );
imgData.data.set( readbackBuffer ); // throws error
ctx.putImageData( imgData, 0, 0 );

I get the following runtime error:

webgpu_sandbox.html:85 Uncaught (in promise) RangeError: offset is out of bounds