Support for up to 64 GPU blendshapes/morphTargets required for Pro-Level Facial Animation

[bhouston]

Is your feature request related to a problem? Please describe.

High fidelity facial animation requires a lot of blend shapes. For example, here is a standard blend shape set from Apple for facial animation that they can extract from their cameras. It has 57 blend shapes.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation

It was utilized in this demo that drove a ReadyPlayerMe avatar in Three.js via the Apple ARFaceAnchor blend shapes:

https://twitter.com/seflless/status/1381313468926676997

You can see here that Three.js currently limits morphTargets (a less common name for blend shapes) to just 4:

https://github.com/mrdoob/three.js/blob/dev/src/renderers/shaders/ShaderChunk/morphtarget_vertex.glsl.js

Describe the solution you'd like

It would probably be best to look are removing a hard upper limit to the number of blend shapes, or to bring it to 64 (which seems like a safe number.)

Describe alternatives you've considered

We have currently implemented unlimited blend shapes via the CPU rather than the GPU. This allows for unlimited number of blend shapes but it is significantly slower.

Additional context

I've asked @selfless to join this conversation to describe how he implemented it so his approach could inform how this will become standard in the library.

[Mugen87]

Some code for this was once shared in #15556 (comment).

[bhouston]

Here are some options:

• Have a scheme that can calculate the blend shapes on the CPU. Basically an option to switch to CPU blend shape evaluation.
• I think the GPU limitations are severe and could not scale up to 64 vertex and normal attributes. This means that there may need to be a way around using attributes for these.

Methods around the GPU limitation on attribute numbers:

• Use a data texture: https://www.robg3d.com/projects/texture-based-blendshapes-in-dx9/. Probably requires an FP32 or FP16 texture extension?
• Doing some research I see this paper from NVIDIA that uses the "StreamOut" feature of DirectX 10 to do this: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits. I am unsure if there is an equivalent in WebGL -- but I doubt it.
• Could one do a special pre-render loop where one updates the blend shape geometries using a transform feedback shader? One could have a vertex shader that could run multiple times, say with 8 or 16 vertex blend shapes each time and it would accumulate the results? This would be a transform feedback shader: https://developer.mozilla.org/en-US/docs/Web/API/WebGLTransformFeedback. This is just a idea -- I haven't seen evidence it can work, but it may. Transform Feedback shaders are only available in WebGL2.

[bhouston]

@selfless has replied and said that he based his unlimited blend shape method on data textures:

https://twitter.com/BenHouston3D/status/1381585681286377474

[mattrossman]

BabylonJS 5.0 also implemented infinite blend shapes using a texture array, with support limited to WebGL2. See their "how it works" video: https://www.youtube.com/watch?v=LBPRmGgU0PE

[Francois-Laberge-Bose]

Here's my code for reference, including warts and all. I'll come back with clearer code later, at work now. Again, no support on at least iOS because of the lack of support for float textures (THREE.RGBFormat, THREE.FloatType).

Note: I was focused on getting ReadyPlayerMe avatars running in browser. You'll see the number 63 hardcoded in this one, but that's unnecessary obviously.

See video

im-todd-so-blue.mov

const loader = new GLTFLoader();
// Load a glTF resource
loader.load(
    // resource URL
    avatarURL,
    // called when the resource is loaded
    function(gltf) {
        avatar.current = gltf
        scene.add(avatar.current.scene);

        changeToTextureBasedMorphTargets(avatar.current.scene.getObjectByName("Wolf3D_Head"));
        changeToTextureBasedMorphTargets(avatar.current.scene.getObjectByName("Wolf3D_Teeth"));

        /* Modifications to do
            1. Create texture of all morphTarget points
            2. Remove existing morphtarget related attributes
            3. Disable all morphTarget logic by setting appropriate flags
            4. Replace existing vertex shader morphTarget declarations with our own
          */
        function changeToTextureBasedMorphTargets(skinnedMesh) {
            // Create morphTarget positions texture
            let size = 4096 * 4096;
            let data = new Float32Array(3 * size);
            let vertIndexs = new Float32Array(skinnedMesh.geometry.attributes.position.count);
            //Loop over each original vertex
            let stride = 0;
            let vertexId = 0;
            let min = Number.MAX_VALUE;
            let max = Number.MIN_VALUE;
            for (let v = 0; v < skinnedMesh.geometry.attributes.position.array.length; v += 3) {
                //Loop over all blendshapes
                for (let i = 0; i < skinnedMesh.geometry.morphAttributes.position.length; i++) {

                    let morphAttr = skinnedMesh.geometry.morphAttributes.position[i];

                    //checkMinMax(morphAttr.array, v);

                    //Copy x, y, and z for the given vertex
                    data[stride] = morphAttr.array[v];
                    data[stride + 1] = morphAttr.array[v + 1];
                    data[stride + 2] = morphAttr.array[v + 2];

                    stride += 3;
                }

                vertexId++;
                //Also set vertIndex at v to v which is the vert index
                vertIndexs[vertexId] = vertexId;
            }
            console.log("min: ", min, "max: ", max);

            function checkMinMaxComponent(value) {
                if (value < min) {
                    min = value;
                }
                if (value > max) {
                    max = value;
                }
            }

            function checkMinMax(array, startIndex) {
                checkMinMaxComponent(array[startIndex]);
                checkMinMaxComponent(array[startIndex + 1]);
                checkMinMaxComponent(array[startIndex + 2]);
            }

            // Remove existing morph target code related attributes
            skinnedMesh.geometry.deleteAttribute("morphTarget0");
            skinnedMesh.geometry.deleteAttribute("morphTarget1");
            skinnedMesh.geometry.deleteAttribute("morphTarget2");

            // Add new vert index attribute
            skinnedMesh.geometry.addAttribute('vertIndex', new THREE.BufferAttribute(vertIndexs, 1));

            //CREATE DATA TEXTURE AND PLACE ON SHADER MAT
            let dataTexture = new THREE.DataTexture(data, 4096, 4096, THREE.RGBFormat, THREE.FloatType);
            dataTexture.needsUpdate = true;

            //Disable all morphTarget logic by setting appropriate flags
            skinnedMesh.material.morphTargets = false;
            skinnedMesh.material.morphNormals = false;

            const morphTargetDeclarations = `
            //Data texture
            uniform sampler2D texture0;
            //Blendshape influences
            uniform float morphTargetInfluences[63];
            //Current vertex index
            attribute float vertIndex;
            `;
            const morphTargetVertexShaderCode = `
            
            //Offset used for fixing the x y coordinates on the data texture
            float offset = vertIndex * 63.;
            //Loop over every blendshape
            for(int i=0; i<63; i++) {
              float iFloat = float(i);
              //If influence is 0, lets not waste GPU processing, move on
              if(morphTargetInfluences[i] == 0.) {
                continue;
              }
              //Find the x and y position of the vertex data based on vertex index and blendshape index
              float x = mod(offset + iFloat, 4096.);
              float y = ((offset + iFloat) / 4096.);
              
              //Grab the data at x and y
              vec2 texCoord = vec2(x / 4096.,y / 4096.);
              vec4 data = texture2D(texture0, texCoord);
              
              //Modify the current vertex position with the data found in the texture and the current blendshape influence
              transformed.x += data.x * morphTargetInfluences[i];
              transformed.y += data.y * morphTargetInfluences[i];
              transformed.z += data.z * morphTargetInfluences[i];
            }
          `


            // Replace all existing morph target vertex shader code with our own
            skinnedMesh.material.onBeforeCompile = function(shader) {
                // Add new necessary uniforms
                shader.uniforms.texture0 = {
                    type: 't',
                    value: dataTexture
                };
                // Now when we change the morphTargetInfluences it's linked to show up in the 
                // vertex shader
                shader.uniforms.morphTargetInfluences = {
                    value: skinnedMesh.morphTargetInfluences
                };

                // Replace morphTarget code declarations with our own
                shader.vertexShader = shader.vertexShader.replace(
                    "#include <morphtarget_pars_vertex>",
                    morphTargetDeclarations
                );

                shader.vertexShader = shader.vertexShader.replace(
                    "#include <morphtarget_vertex>",
                    morphTargetVertexShaderCode
                );
            };
        }
    },
    // called while loading is progressing
    function(xhr) {
        console.log((xhr.loaded / xhr.total * 100) + '% loaded');
    },
    // called when loading has errors
    function(error) {
        console.log('An error happened');
    }
);```

[Francois-Laberge-Bose]

Some thoughts on what's next:

• Try encoding the points in regular textures instead
  • This is a challenge as with only 256 bits per channel, if you encode say [-0.5m, 0.5m] from that you get 1m/256 = 7.8mm of resolution, that's almost a centimeter, and artifacts will be noticeable for things like facial animation.
  • I was starting to look at encoding for the exact range of values that were in my readyplayerme avatar, which is much less than a meter. So you could make the encoding be based on the actual value range in a particular model, but this means two passes over the blendshapes to first calculate min/maxes. Again, pre generating get away from that, but then you are dealing with non standard glb files. Perhaps an extension or something special can be put in the file that three.js or others are aware of.
• Make sure texture size is the minimal it can be
• (Optional) Come up with script to pre-process a glb you know you are going to load, to pre-generate the lookup texture and strip out the morphTarget points from the file

[seflless]

A little self promotion here, but if you are interested in an upcoming tool/app for streaming iOS/standalone-VR to the web, follow this account, where I'll be sharing the progress, and announcements about the beta. The above was created for this work:
https://twitter.com/keypoints_app

[bhouston]

@seflless very awesome! You said that iOS doesn't support float textures, but my iPhone does. it supports OES_texture_float, OES_texture_half_float. But can it access them in the vertex shader? Also Safari supports these extensions. I believe that nearly every modern device supports these two extensions now - or they support WebGL2 which requires these extensions I believe.

[seflless]

I'm going to have to rerun my app and debug it to share the error. I swear I also confirmed it was not going to be supported on iOS. I could not find any links in my notes about this though. Looks like I broke the code I deployed, so I'm going to have to fix something before I can show a reproduce-able error.

[bhouston]

I ran the web app here because you shared the URL in the video. I got XR errors on attempt to load in Safari anywhere. It worked on Chrome because it supports the XR interfaces.

[Francois-Laberge-Bose]

Yeah, I'm deploying a fix for that. I wasn't checking for the existence of navigator.xr before calling it. I'm supporting WebXR previewing of the stream, and using VR for head+hand tracking capture.

[seflless]

Ok the unrelated bug is fixed, I get the following error:

THREE.WebGLRenderer: EXT_color_buffer_float extension not supported.

More about what the EXT_color_buffer_float extension gets you: https://developer.mozilla.org/en-US/docs/Web/API/EXT_color_buffer_float

This is why I believed it wasn't possible. MDN has it as not supported by macOS or iOS safari.
https://developer.mozilla.org/en-US/docs/Web/API/EXT_color_buffer_float#browser_compatibility

Ok, here you go to see for yourself:
http://keypoints.vercel.app/