GLTFLoader: record association between glTF elements and Three.js objects

[cdata]

Note: I don't want to bury the lede for this request, so I moved a brief summary to the top of the issue. Please refer to the second and third headings for a more detailed background and rationale!

Proposal

When loading models with GLTFLoader, it would be nice if we could retain associations between glTF scene graph elements and the Three.js objects that correspond to them. I briefly discussed this problem with @donmccurdy and he suggested that one of the following approaches might be acceptable:

1. Decorate Three.js objects via userData e.g., material.userData.gltfMeta.index = 3
2. Expose a lookup table somewhere e.g., parser.getDefinition( material: THREE.Material ): GLTF.Material

What do others think about this problem? Would you be willing to consider a change that makes either of the suggested enhancements?

Background

Three.js' GLTFLoader currently produces an artifact that contains a Three.js scene graph as well as a GLTFParser instance that holds the original, deserialized glTF.

The GLTFParser holds associations between the elements in the original glTF and the Three.js scene graph. These associations are expressed both by GLTFParser.prototype.getDependencies and the cache object on the GLTFParser instance.

In some cases, the relationship between the original glTF element and the associated Three.js objects is obscured and cannot be trivially re-established. For example, materials associated with a skinned mesh are cloned, and the resulting clones are cached against a key that cannot be reconstructed easily:

three.js/examples/jsm/loaders/GLTFLoader.js

Lines 2118 to 2127 in 648c4bb

	cachedMaterial = material.clone();
	if ( useSkinning ) cachedMaterial.skinning = true;
	if ( useVertexTangents ) cachedMaterial.vertexTangents = true;
	if ( useVertexColors ) cachedMaterial.vertexColors = true;
	if ( useFlatShading ) cachedMaterial.flatShading = true;
	if ( useMorphTargets ) cachedMaterial.morphTargets = true;
	if ( useMorphNormals ) cachedMaterial.morphNormals = true;
	this.cache.add( cacheKey, cachedMaterial );

Description of the problem

Consider a model editor that allows a user to:

1. Load a glTF with any shape
2. Interactively change the value of any field of any element of the glTF
3. See the result of the change rendered in real time
4. Export a new glTF

In an ideal world, the editor should faithfully re-export the model with its original hierarchy and shape. The only meaningful change in the exported file should be the one made interactively by the user.

In the pursuit of this goal, it is very useful (and probably required) that an association between the source glTF scene graph and the Three.js scene graph be made, so that for any change we make to a glTF element, we can reflect that change in the rendered Three.js scene in real-time without completely re-loading the glTF.

Some of the information required to correctly correlate the glTF scene graph with the Three.js scene graph is lost or obscured, and this makes the above described editor scenario very difficult (perhaps impossible) to achieve.

Three.js version

• Dev
• r115
• ...

Browser

• All of them
• Chrome
• Firefox
• Internet Explorer

OS

• All of them
• Windows
• macOS
• Linux
• Android
• iOS

Hardware Requirements (graphics card, VR Device, ...)

[cdata]

If this request is broadly acceptable, I would like to suggest that approach (1) suggested by @donmccurdy seems preferable because it will be easier to retain associations when making clones of the scene graph.

[mrdoob]

@cdata approach (1) sounds good to me 👌

[donmccurdy]

@drcmda and @fms-cat: I think you've both asked about variations of this in the past. Would either of these approaches be helpful for you?

[cdata]

One hitch with approach (1) is that some of the objects created by Three.js do not have a `userData` property. For example, `AnimationClip` and `BufferAttribute`. Would it make sense to add `userData` to these? Or perhaps this is a good reason to go with option (2) instead?

…

On Fri, May 1, 2020, 5:45 PM Don McCurdy ***@***.***> wrote: @drcmda <https://github.com/drcmda> and @fms-cat <https://github.com/FMS-Cat>: I think you've both asked about variations of this in the past. Would either of these approaches be helpful for you? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19257 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB2TU6CCO6AO5CNLCSCEC3RPNUKDANCNFSM4MUBEYTA> .

[drcmda]

i would most definitively prefer a lookup table. react-three-fibers useLoader adds "nodes" and "materials" to the gltf data. with these two the entire graph can be represented.

this enables full automation, for instance this is how a file from sketchfab looks like: https://codesandbox.io/s/romantic-tharp-0buje?file=/src/Model.js

in plain threejs it could look like this:

new GLTFLoader().load(url, ({ nodes, materials }) => {
  nodes.cube.receiveShadow = true
  materials.metal.roughness = 0.5
  ...
})

users now could go ahead and build their own graphs:

const mesh = new THREE.Mesh(nodes.cube.geometry, materials.metal)
scene.add(mesh)

as for userData, i think object mutation isn't the best choice imo. i don't like mutation in general, and it still needs traversal in the end, which is the very thing that makes handling gltf so hard in threejs. if you add a lookup table with gltf indices that would be nice. official support for nodes/materials would be welcome, too - then i can remove it from r3f.

if you go for a index table alone, pretty please dont name it "nodes" 👀

[0b5vr]

My original intention of the idea 1 was to load the custom material from the glTF and that still should be achieved using plugins.
This one sounds like a totally different purpose and also a valid idea 👍

End developer have to delete the userData field when they try to clone a gltf derived materials in this case, and that sounds weird a bit 🤔

[donmccurdy]

One hitch with approach (1) is that some of the objects created by Three.js do not have a userData property. For example, AnimationClip and BufferAttribute.

I'm not inclined to include every possible object type, until we have concrete reasons to do so. If the solution happens to handle everything easily, neat, but it can probably wait.

new GLTFLoader().load(url, ({ nodes, materials }) => {
nodes.cube.receiveShadow = true
 materials.metal.roughness = 0.5
...
})

...as for userData, i think object mutation isn't the best choice imo

I'd be OK with exposing some kind of lookup table, either on GLTFLoader's result object or on the GLTFParser it already returns. It probably should not rely on unique names, which are not guaranteed, but indices or a Map would work.

if you go for a index table alone, pretty please dont name it "nodes"

I don't understand — the name conflicts with something you need?

[drcmda]

It probably should not rely on unique names, which are not guaranteed, but indices or a Map would work.

the names though is what makes it usable - and typesafe - , otherwise to find out where "robotLeg" is you traverse again, as opposed to:

const { nodes: { robotLeg } } = await new GLTFLoader<Robot>().loadAsync(url)
robotLeg.receiveShadows = true

someone could hack into a gltf with a text editor and use one name multiple times, it isn't possible in modeling software like blender and unnamed objects are named by three with an additive index. all it would do is overwrite a named slot, no harm done. but we're forcing all users to traverse in order to find back their assets because of it.

I don't understand — the name conflicts with something you need?

if you do decide to add something that's working differently than a name:value collection, please consider naming it something else than "nodes" and "materials" if it's in any way possible. i know that's a lot to ask for, but it would throw us into major breaking changes.

[cdata]

@drcmda @donmccurdy I have taken to referring to the glTF descriptors as glTF "elements." Not sure if that helps the discussion, but worth considering in case you want another word to refer to those things collectively 💁

It sounds like a look-up table is the preferred approach. I took a quick stab at this last week. Perhaps I could put up a PR for considering?

I'm not inclined to include every possible object type, until we have concrete reasons to do so.

Is there a reason we should keep in mind for why you might be disinclined to include some object types? Subtle complexity or anticipated changes in the implementation, perhaps?

[donmccurdy]

someone could hack into a gltf with a text editor and use one name multiple times, it isn't possible in modeling software like blender and unnamed objects are named by three with an additive index.

We do see objects with duplicate names periodically; there have been multiple bug reports against three.js as a result. That's solvable: I think GLTFLoader should automatically rename duplicates. But more importantly, mapping from a unique name (possibly generated by GLTFLoader) to a three.js object does not provide the mapping back to the glTF definition that @cdata presumably needs. You would need to also return an annotated version of the original glTF JSON, with new names. 😕

Is there a reason we should keep in mind for why you might be disinclined to include some object types?

Mostly I'm nervous of modifying three.js' API just to match glTF, if that change doesn't also bring some real benefit to the API itself. And, there are various objects (samplers, textures vs images) that exist in glTF but don't obviously map to a three.js object in the final scene. Mapping nodes, meshes, and materials is a much more tractable problem, but still has plenty of edge cases.

[cdata]

I'm definitely in agreement about not using names to map these things. The glTF spec is really explicit about not being able to rely on name uniqueness:

Any top-level glTF object can have a name string property for this purpose. These property values are not guaranteed to be unique as they are intended to contain values created when the asset was authored.

Source: https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#indices-and-names

Mostly I'm nervous of modifying three.js' API just to match glTF, if that change doesn't also bring some real benefit to the API itself.

FWIW all that matters to me at the end of the day is that it is possible to cross-associate between the two hierarchies. The relative stability of this association over time is not nearly as important. That said, I appreciate that any nature of association will be considered an implicit contract in the the API.

To look at this somewhat differently, if it were possible for a GLTFLoader user to participate in the parsing process, this might give Three.js more conceptual flexibility and still enable someone like me to satisfy the original stated goals. The GLTFParser already has a fairly well-factored implementation that separates loading of each distinct part of the glTF:

three.js/examples/jsm/loaders/GLTFLoader.js

Lines 1591 to 1665 in 51c70b6

	/**
	* Requests the specified dependency asynchronously, with caching.
	* @param {string} type
	* @param {number} index
	* @return {Promise<Object3D|Material|THREE.Texture|AnimationClip|ArrayBuffer|Object>}
	*/
	GLTFParser.prototype.getDependency = function ( type, index ) {
	var cacheKey = type + ':' + index;
	var dependency = this.cache.get( cacheKey );
	if ( ! dependency ) {
	switch ( type ) {
	case 'scene':
	dependency = this.loadScene( index );
	break;
	case 'node':
	dependency = this.loadNode( index );
	break;
	case 'mesh':
	dependency = this.loadMesh( index );
	break;
	case 'accessor':
	dependency = this.loadAccessor( index );
	break;
	case 'bufferView':
	dependency = this.loadBufferView( index );
	break;
	case 'buffer':
	dependency = this.loadBuffer( index );
	break;
	case 'material':
	dependency = this.loadMaterial( index );
	break;
	case 'texture':
	dependency = this.loadTexture( index );
	break;
	case 'skin':
	dependency = this.loadSkin( index );
	break;
	case 'animation':
	dependency = this.loadAnimation( index );
	break;
	case 'camera':
	dependency = this.loadCamera( index );
	break;
	case 'light':
	dependency = this.extensions[ EXTENSIONS.KHR_LIGHTS_PUNCTUAL ].loadLight( index );
	break;
	default:
	throw new Error( 'Unknown type: ' + type );
	}
	this.cache.add( cacheKey, dependency );
	}
	return dependency;
	};

It might be sufficient to support a callback that is invoked at the tail-end of loading each dependency. For example:

GLTFParser.prototype.getDependency = function ( type, index ) {
  // ...

  if ( ! dependency ) {
    switch ( type ) {
      case 'scene':
        dependency = this.loadScene( index );
        break;
      // ...
    }
    
    if ( this.dependencyCallback ) {
      this.dependencyCallback( type, index, dependency );
    }
  }

  // ...
};

The nice thing about this approach is that it is simple to implement and the structure of glTF is the primary driver of the API (rather than the ways in which the glTF is interpreted in terms of Three.js). This perhaps conceptually lightens the burden on Three.js implementation. WDYT @donmccurdy ?

[cdata]

Although I should note that the use case is orthogonal to the current feature request, it is worth considering that the dependencyCallback approach could help folks when prototyping support for experimental or soon-to-be-ratified glTF extensions. To support this case, it might make sense for the dependencyCallback to be designed as a mapping function.

This isn't to suggest we should support this, but it is an interesting capability that is potentially unlocked with the right design. The idea came up in discussion with some folks from 3D Commerce today while we were speculating about how we might experiment with KHR_materials_variant support in Three.js without forking anything.