hand-controls misaligned with HTC Vive controllers in 1.4.2

[wyzwon]

Description:
As of commit 5b29ac2, the hands no longer line up with the HTC Vive controllers.

This is the log for the specific commit in which the issue first appears:

hand-controls: rotates models to match WebXR definition (fix #4848) (#5261)

It is a small commit that only changes handModelOrientation. It seems the Vive is not in agreement with the new WebXR standard. Is this an issue that exists at the level of SteamVR and their WebXR implementation? If so, maybe a workaround is possible in the mean time?

Note: Due to an issue with the build bot at the time, you won't see the issue in the cdn.jsdelivr.net link supplied for commit 5b29ac2 or any other commit before 7050d7b (the last April 19th commit). You'll have to remake the build yourself if you want to try those commits for any reason.

• A-Frame Version: 1.4.2 (5b29ac2 and up)
• Platform / Device: gen 1 HTC Vive / Windows 10 / Steam VR 1.26.3 (beta) / Chrome 114

[mrxz]

I was able to reproduce the issue with Valve Index controllers. The problem is entirely in A-Frame due to inconsistencies in how it handles various controls. When using hand-controls, there is a chain of dependencies as follows:

• hands-controls
• oculus-touch-controls, vive-controls, etc... (annoyingly enough it's missing valve-index-controls)
• tracked-controls
• tracked-controls-webxr (or in case of WebVR tracked-controls-webvr)

The hands-controls assumes that the object is in gripSpace in which case the rotation added in commit 5b29ac2 is correct. However, the default space used by tracked-controls is targetRaySpace. Literally only oculus-touch-controls deviates from the default and configures tracked-controls to be gripSpace, all other controls components result in targetRaySpace, meaning the entire pose of the hand (position and rotation) is mostly bogus.

As a workaround you can manually override the space for tracked-controls:

<a-entity hand-controls="hand: left" tracked-controls="space: gripSpace"></a-entity>

[wyzwon]

Using tracked-controls="space: gripSpace seems to put their center of mass back in line with the physical controllers but the hands are still pitched 90 degrees forward relative to their old alignment.

[mrxz]

(...) the hands are still pitched 90 degrees forward relative to their old alignment.

Some screenshots might help, though it's probably hard to see how they should be orientated.

It's worth noting that the hand model is just straight and it seems neither orientations are perfect. At least for me on both Quest 2 and Valve Index the additional rotation does result in better orientation for the fingers. For example, if I stick out both my index fingers and try to make them touch, it lines up decently. Sticking out my thumbs also go in the right direction. But the wrists seem to "stick out" slightly upwards, as they don't angle down like they do in real life.

Without the additional rotation, the wrist seems to line up better, but the finger orientations no longer match. If I stick out my thumbs, the model does reflect this "gesture", but the thumbs direction are 90 degrees off from the actual direction my thumbs are pointing, same with the index fingers.

Does this align with what you're seeing with the Vive controllers?

[wyzwon]

1.4.1 and below orientation

1.4.2 orientation (with tracked-controls="space: gripSpace")

The new orientation beyond being an unexpected change, is way too clunky for a controller like the Vive controllers which are meant to be held more like a wand. The model's wrist sticks out of the base of my thumb in this rotation.

I was quite happy with how things were, it felt natural to grab things that way.

Fwiw, Valve takes a somewhat middle position based on their update to The Lab which gave the player robot hands. In their setup, the loop thing is about level with the index finger, and the handle passes through the edge of the wrist opposite the thumb.

[mrxz]

Thanks for the images. It's effectively the inverse problem a lot of older VR games have that were made with the VIVE wands when played with Index controllers. The tricky part is that WebXR doesn't provide a whole lot of information. The "rod" in grip space is given, but no clue as to how the user would hold this "rod" (as a wand, straight on or somewhere in between)

Wondering what the best approach for this is, but as a crude workaround you can manually set the rotation of the hand mesh after it has been loaded. Something like the following should do the trick:

const mesh = handControlsEl.getObject3D('mesh');
if(mesh) {
    mesh.rotation.x = 0;
} else {
    handControlsEl.addEventListener('object3dset', function waitForMesh(e) {
        if(e.detail.type !== 'mesh' || e.target !== handControlsEl) { return; }
        setTimeout(() => handControlsEl.getObject3D('mesh').rotation.x = 0, 0);
        handControlsEl.removeEventListener('object3dset', waitForMesh);
    });
}

Note the setTimeout is there since the hands-controls component first sets the mesh and then immediately updates the rotation, so without it the rotation would be overwritten by the component.
(Alternatively you can make your own copy of hand-controls with the relevant changes made directly to it)

[dmarcos]

<a-entity hand-controls="hand: left" tracked-controls="space: gripSpace"></a-entity>

Should probably consolidate all in controllers in gripSpace?

[mrxz]

<a-entity hand-controls="hand: left" tracked-controls="space: gripSpace"></a-entity>

Should probably consolidate all in controllers in gripSpace?

From a consistency pov I agree, it's weird that only the oculus-touch-controls deviate from the rest. However, this might not be a simple change. It would impact all usages of xyz-controls, meaning controller models no longer align and any experience attaching raycasters under the assumption that they are in targetRaySpace would break.

Given that it's the hand-controls component that expects gripSpace to orientate the hand mesh, it might be an option to have it explicitly set this?

[wyzwon]

Do we have a way to distinguish the wands vs the index controllers? It would be ideal to detect and handle them differently or one will always be off (I assume anyways, I don’t have an index controller to try).

I could always add a manual switch in a settings tab if it came to it, but that’s a last resort.

[dmarcos]

FWIW we have separate components for vive-controls and valve-index-controls

[wyzwon]

That is nice and useful, it does solve the problem for something like an isolated museum exhibit where you know the hardware. I wouldn’t want to limit an online experience though.

[mrxz]

Under the hood the hand-controls uses these individual components, which you can check to see which one is actually active. For example:

if(handControlsEl.components('vive-controls').controllerPresent) {
   // Do something only for Vive controllers
}

[wyzwon]

Does “Vive-controls” mean just the wand? or does it trigger for index controllers as well? If that does distinguish them, shouldn’t that mean this is fixable under the hood?

I would certainly put in some sort of reactive correction if it worked and I need to for my project, but it would be nice to have it fixed everywhere, or are we considering the new orientation with wands to be the intended standard?

[mrxz]

Does “Vive-controls” mean just the wand? or does it trigger for index controllers as well?

It shouldn't trigger for the index controllers, but it does currently trigger for all HTC controllers (specifically the ones that prefix match htc-vive, like for example htc-vive-cosmos). A more rigorous check for the HTC Vive Wand would be something like:

handControlsEl.components['tracked-controls-webxr'].controller.profiles[0] === 'htc-vive'

(The Vive Wands should have the exact profile id "htc-vive")

If that does distinguish them, shouldn’t that mean this is fixable under the hood?

A "fix" can't really be applied to the vive-controls component directly. When using the vive-controls separate from hand-controls it should show you a 3D model of the controllers and be orientated in targetRaySpace. (I assume that in this configuration the controller models line up correctly?).

So only in the context of hand-controls does this offset make sense. Ideally we avoid the situation where the component needs different logic for each type of controller in addition to the information it gets from WebXR about the controller. I think the situation could perhaps be improved by using the gripSpace to know where the fingers/palm go and the (global) orientation to know where the wrist (likely) goes. Aligning the hand with those should make the behaviour more consistent across controllers (current and future). 🤔

[wyzwon]

I definitely agree it wouldn't be ideal. It reminds me of the (thankfully mostly) gone days of browser identification and browser specific hack fixes, though if it ended up being the only way, I would consider it better than leaving the hands impractically bent for certain controllers. Hopefully your idea works out though.

I wonder if the group behind the standard is aware of this limitation?

When using the vive-controls separate from hand-controls it should show you a 3D model of the controllers and be orientated in targetRaySpace. (I assume that in this configuration the controller models line up correctly?).

Yes this does display correctly; in fact to get those pictures above I just stuck a pair of vive-controls components in the scene alongside the hand-controls.

[wyzwon]

So I opened up hand-controls and took a look around inside. I noticed that there is a "isVive" flag.

@mrxz I know you had this to say about the subject:

Ideally we avoid the situation where the component needs different logic for each type of controller in addition to the information it gets from WebXR about the controller.

but given that we are already doing the User-Agent string thing anyways, and a proper fix appears far off (out of our scope?), maybe we should consider just adding a re-adjustment for Vive wand controllers specifically right now to get things back to at least working order in the mean time.