BatchedMesh: Proposal

[donmccurdy]

Currently the APIs available for reducing draw calls include:

• InstancedMesh
• BufferGeometryUtils.mergeBufferGeometries([a, b, c, ...])

I think we might be able to solve some common difficulties with reducing draw calls, even where the objects to be rendered are not instances of the same thing. I've floated the idea in a couple of earlier comments (#19164 (comment), #18918 (comment)) but wanted to open a separate issue for it.

Calling this BatchedMesh for now, it's meant to be a way of (1) merging, (2) rendering, and (3) updating a group of objects that would otherwise be drawn as separate Mesh instances. The objects must all be the same primitive type (i.e. triangles), must share a material, and must have compatible vertex attributes (e.g. if one has normals, they all have normals). With those requirements we could offer an API like:

// Create BatchedMesh with max vertex and triangle counts to support. If 
// geometry is not indexed, maxTriangleCount is optional. The 'template'
// geometry could just be the first geometry we'll add later — it isn't stored,
// but is used to determine which vertex attributes to allocate.
const batchedMesh = new BatchedMesh( templateGeometry, material, maxVertexCount, maxTriangleCount );

// Append geometry, up to max vertex and triangle limit.
const id1 = batchedMesh.add( geometry1, matrix1 );
const id2 = batchedMesh.add( geometry2, matrix2 );
const id3 = batchedMesh.add( geometry3, matrix3 );

scene.add( batchedMesh );

renderer.render( scene, camera );

// Update object position dynamically. For small objects this may be cheaper than
// a separate draw call. For large objects, it may be better to draw the object
// separately than to update it with dynamic batching.
batchedMesh.setMatrixAt( id1, otherMatrix );

renderer.render( scene, camera );

This offers a few advantages over mergeBufferGeometries(). First, you get an ID that lets you remove or modify each geometry later. Second, we can potentially do much faster raycasting (we know the bounding box and transform of each member geometry) and return that ID as a result. Third, we can do frustum culling (maybe?) by updating the index buffer.

[takahirox]

Sounds like it fits to WebGL_multi_draw extension.

https://developer.mozilla.org/en-US/docs/Web/API/WEBGL_multi_draw

Advantage:

• Visibility handling (invisible due to frustum culling or .visible=false) may be easier. You can easily specify which objects in BatchedMesh should be rendered by controlling furstList, countsList, and drawCount of multiDrawArraysWEBGL().
• You can use gl_DrawID in shader. So if you use uniform array, you can apply different uniform parameter for each object in BatchedMesh. It would be more flexible, for example you can apply different material color for each object, and more performant, for example you will no longer need to directly update positions in setMatrixAt() in CPU.

Disadvantage:

• Some major browsers, eg: Firefox and Safari, don't support the extension yet
• If we use the uniform array approach, the maximum uniform array size can limit the number of objects which can be added to BatchedMesh.

[donmccurdy]

Sounds like it fits to WebGL_multi_draw extension.

Oh even better! Do you know if there's signal from Firefox or Safari on plans to support yet? If it's not on the way I would probably lean toward doing a first implementation without it, but trying to keep a compatible API so we can upgrade later. This also gives the advantage that we can implement it in examples/jsm/* without core changes just yet.

[donmccurdy]

and more performant, for example you will no longer need to directly update positions in setMatrixAt() in CPU.

This would be really helpful... And it saves us from having to keep an extra (un-transformed) copy of vertex data in memory. If I understand correctly, dynamic batching in popular game engines will not do this, but instead rewrites vertices on the CPU, I'm not sure why that would be if this extension is widely availlable. 🤔

[jbaicoianu]

@donmccurdy I did some experiments with this idea a few years back, it's definitely an improvement if you have lots of mostly-static geometry in the scene.

One trick I found that made it easier to work with is to use THREE.Mesh instead of geometry + matrix, and then the batchedMesh.add() calls return a proxy object, THREE.MergedObject or something similarly-named, which lets you continue to use .position .rotation etc. as if it were still a separate standalone object. This proxy object can be set up so that changes are automatically carried through to the merged object's combined buffers, so the developer can use the same code for both merged and unmerged cases:

const batchedMesh = new BatchedMesh( templateGeometry, material, maxVertexCount, maxTriangleCount );

// Append meshes, up to max vertex and triangle limit.
const obj1 = batchedMesh.add( mesh1 );
const obj2 = batchedMesh.add( mesh2 );
const obj3 = batchedMesh.add( mesh3 );

function update(dt) {
  obj1.position.set(Math.sin(performance.now() / 1000), 0, 0);
  obj1.rotation.y += dt;
  // etc
}

[takahirox]

I have been wanting to introduce WEBGL_multi_draw extension to Three.js but I gave up before because the existing Three.js API doesn't really fit to the extension. So I'm very positive to introduce a new Three.js API (BatchedMesh) which may be fitting to the extension.

@donmccurdy

Do you know if there's signal from Firefox or Safari on plans to support yet?

The WebGL_multi_draw support ticket for Firefox has been made but it isn't first priority (P2), so I don't think at least it lands soon.

https://bugzilla.mozilla.org/show_bug.cgi?id=1536673

I have no idea about Safari.

I would probably lean toward doing a first implementation without it, but trying to keep a compatible API so we can upgrade later.

That sounds good to me. Even if we use the extension, we will need fallback path until all major browsers support the extension.

Even without the extension, we may be able to use the uniform array approach by adding an extra vertex data having id. It may be a bit memory costly tho.

And it saves us from having to keep an extra (un-transformed) copy of vertex data in memory.

Yeah. And we will not need to transfer updated positions to WebGL, it is another performance advantage.

If I understand correctly, dynamic batching in popular game engines will not do this, but instead rewrites vertices on the CPU, I'm not sure why that would be if this extension is widely availlable. 🤔

Interesting... I guess updating big uniform array every frame might be costly or they want scalability (the maximum uniform array size limits the number of objects)?

How can I know the maximum uniform array size? Max Fragment/Vertex Uniform Vectors?

https://webglreport.com/?v=2

And I'm wondering how many objects per one BachedMesh we will want to support.

@jbaicoianu

Personally I prefer your API idea because it would be more user friendly. But coming up some thoughts in my mind, for example what obj1/2/3.clone() should mean? Should we restrict some methods for that usage?

Update: I prefer passing geometry rather than mesh on this API idea because meshes can have different materials.

const obj1 = batchedMesh.add( mesh1 );

[donmccurdy]

Maybe two ways to approach the proxy objects:

• (a) they inherit from Object3D, and can be added to the scene graph and manipulated there
  • More overhead, we're paying the full cost of the scene graph traversal, but it's a very convenient API. Will hit performance cliffs for large object counts.
• (b) they are just simple objects that expose proxied position/rotation/scale/matrix properties
  • We get notified when something changes and overhead is lower, but there's no scene graph.
  • I think this is probably a bad compromise. Users can't reuse code from un-batched objects, it's inconsistent with the InstancedMesh API, and still has per-object allocation overhead.

Perhaps relevant, Babylon.js' instancing API offers a choice of proxy objects and "thin" instancing (without proxies), and the latter improves performance "drastically" for large counts. I don't necessarily want to jump straight to offering two APIs, but do wonder if there are lessons learned here.

Also – the performance difference between "merged" and "naive" on this example is... not much. Perhaps because it's the same geometry? Both around 25-30 FPS on my machine. Without a better understanding of why that is, it might be better to skip all the avoidable scene graph overhead for now, until we can prove that we've got a performance win, and then make the API easier to use from there.

[drcmda]

This proxy object can be set up so that changes are automatically carried through to the merged object's combined buffers, so the developer can use the same code for both merged and unmerged cases:

why not simply using mesh and that's it? i wonder why we need proxies or object ids at all? once the meshes are added the batchedmesh instance has full access to all its children and can read out the transforms whenever it wants, or maybe i just don't understand the limitations that it would cause. but in any case, i would always prefer an api surface that can be expressed imperatively and declaratively and that is something three's own object3d/mesh does exceptionally well.

const batchedMesh = new BatchedMesh(templateGeometry, material, maxVertexCount, maxTriangleCount)

batchedMesh.add(mesh1)
batchedMesh.add(mesh2)
batchedMesh.add(mesh3)
...
function update(dt) {
  mesh1.position.set(Math.sin(performance.now() / 1000), 0, 0)
  mesh1.rotation.y += dt
}
...
batchedMesh.remove(mesh1)
...

edit:

i take it the main problem with meshes is the material. batched mesh already sets one, so repurposing mesh like that probably is unwanted. could an intermediate object maybe help? if this isn't thrown into a batch it could maybe render black or not at all.

class MeshInstance extends THREE.Object3D {
  constructor(geom) {
    super()
    this.geometry = geom
    // this.needsUpdate = false → is this the reason we're talking proxies? aren't flags a precedent?
  }
}

[jbaicoianu]

why not simply using mesh and that's it? i wonder why we need proxies or object ids at all? once the meshes are added the batchedmesh instance has full access to all its children and can read out the transforms whenever it wants, or maybe i just don't understand the limitations that it would cause

Using a proxy object (either a literal ES6 Proxy or just some intermediate with getters/setters) can help with performance. There isn't really an easy built-in way to observe an object for changes and only run some code (eg, reprojecting some vectors) for objects when they change position, orientation, scale, or any other properties that could be expressed as uniforms or attributes - using a proxy object you can present the same interface, but set dirty flags which can then be checked each frame to know whether that object needs updating in the batched mesh.

but in any case, i would always prefer an api surface that can be expressed imperatively and declaratively and that is something three's own object3d/mesh does exceptionally well.

Part of my experiment involved exposing this in a declarative way for JanusWeb, if I recall it just involved wrapping the objects I wanted to merge in a <CombinedMesh> container, and that automatically merged its children into itself as they loaded

<CombinedMesh>
  <Object src="model1" pos="1 2 3" />
  <Object src="model2" pos="0 0 1" />
  ...etc...
</CombinedMesh>

i take it the main problem with meshes is the material. batched mesh already sets one, so repurposing mesh like that probably is unwanted. could an intermediate object maybe help? if this isn't thrown into a batch it could maybe render black or not at all.

As proposed yeah, the BatchedMesh object takes a material as the argument, which would make the mesh.material property superfluous. I can imagine an additional BatchedMeshGroup object which doesn't take a material arg, and when you add a mesh (or a hierarchy of objects) to it, it could extract all the unique materials, and create a BatchedMesh object for each one.

[looeee]

Also – the performance difference between "merged" and "naive" on this example is... not much. Perhaps because it's the same geometry? Both around 25-30 FPS on my machine.

There's a big difference on my machine. When I put the slider to full it's 60fps on both instanced and merged, 16fps on naive.

[boytchev]

I'm little late to the party.

Here is a set of ideas (not necessarily mutually compatible)

• Skipping the templateGeometry and using the first added geometry as template. Then, all other missing attributes could be set to a default value, all other extra attributes will be ignored. This means, that the actual internal construction is done after the first added geometry.

new BatchedMesh( material, maxVertexCount, maxTriangleCount );

• Skip maxVertexCount and maxTriangleCount. Then the internal construction is delayed just before the first render (after it adding new geometries should be either forbidden, or should cost regeneration of the internal structures)

new BatchedMesh( material );

• Replace templateGeometry with an array of actual geometries, the first one is used as template.

new BatchedMesh( arrayOfGeometries, material );

• Add a mesh (i.e. its geometry and its matrix, but not its material). Yes, I know it has been discussed in previous posts.

const id = batchedMesh.add( mesh );

This might be a shortcut of:

const id = batchedMesh.add( mesh.geometry, mesh.matrixWorld );

• Normalized ids. Instead of some id, return consecutive indices - 0, 1, 2, 3, ... This means, that BatchedMesh must handle internally the mapping from indices to original ids. This will make it easy to do something like this:

for( var i=0; i<n; i++) batchedMesh.setMatrixAt( i, otherMatrix );

• This is not an idea, just an observation. BatchedMesh is somehow the opposite of BufferGeometry.groups. The first is combining geometries and drawing them in one call; the second is splitting a geometry and drawing each part in a separate call.
• I did not understand how to access the other properties of individual geometries (like bounding box and sphere), or methods (like recalculating normals)
• Should there be a method for unbatching a batched mesh into separate meshes; or converting it to an ordinary mesh (by merging geometries)?

[donmccurdy]

Skipping the templateGeometry and using the first added geometry as template. Then, all other missing attributes could be set to a default value, all other extra attributes will be ignored.

Many vertex attributes do not have meaningful default values (e.g. UVs, normals, tangents), and because of how WebGL semantics work we can't really ignore attributes for specific parts of the batch. I'm hoping that being strict about these requirements will create fewer surprises for users than hiding them behind leaky workarounds. I'm not 100% set of having the "template geometry" argument, but we do get complaints when a subclass does not accept the same initial arguments as its parent class (geometry, material in this case).

Skip maxVertexCount and maxTriangleCount. Then the internal construction is delayed just before the first render (after it adding new geometries should be either forbidden, or should cost regeneration of the internal structures)

Note that these are maximum vertex/index counts, they don't have to be exact. I think being able to preallocate a certain amount of space for adding/removing objects in the batch — without incurring cost of re-rebuilding — is an important benefit. InstancedMesh takes a similar approach.

Normalized ids. Instead of some id, return consecutive indices - 0, 1, 2, 3, ... This means, that BatchedMesh must handle internally the mapping from indices to original ids.

Hm this is a good point. We probably do want users to be able to iterate over the objects. But we also want stable IDs, and if an object is removed from the batch, would that invalidate higher indexes? Unlike with InstancedMesh, we actually could support a sparse index list here, e.g. having objects at indexes 1-4 and 6-10, and "nothing" at index=5, could be supported if we wanted.

I did not understand how to access the other properties of individual geometries ...

I don't think we can support every feature and use case of BufferGeometry here, e.g. if you need to recompute normals it would be best to keep the original BufferGeometry around.

---

Two other thoughts. First, .add(...) is already inherited from Object3D so maybe .addGeometry(...) is better here. Second, to support dynamic updates we have to keep the original geometry in memory. That's not ideal if the user doesn't need dynamic updates. Could get around this with a static option:

batchedMesh.addGeometry( geometry, matrix, static = false ); // stores geometry
batchedMesh.addGeometry( geometry, matrix, static = true );  // does not store geometry, setMatrixAt will fail

In either case we can retain the geometry's bounding box to support raycasting.

[donmccurdy]

Alternative, index-based API:

const templateGeometry = geometries[ 0 ];

const batchedMesh = new BatchedMesh( templateGeometry, material, maxVertexCount, maxIndexCount );

batchedMesh.setGeometryAt( 0, geometries[ 0 ], matrix1 );
batchedMesh.setGeometryAt( 1, geometries[ 1 ], matrix2, static = true );

console.log( batchedMesh.count ); // → 2

batchedMesh.setGeometryAt( 1, null );

console.log( batchedMesh.count ); // → 1 (trailing nulls are dropped?)

batchedMesh.setMatrixAt( 0, matrix3 );

^this could be compatible with a proxy- or handle-based API, too, but I guess I like the idea of not requiring proxies to use BatchedMesh, and/or making it possible to build that proxy interface in userland.