Hit test results are fundamentally rays, not poses

[Manishearth]

A hit test result is fundamentally a ray with the origin being the point of intersection, and direction being perpendicular to the plane being intersected with. This is how, for example, Unity represents it.

We're representing it as an XRRigidTransform/XRPose that produces such a ray, i.e. an XRRigidTransform which is capable of transforming the y-axis ray to the desired ray. However, such a transform is not unique (it can put the x and z transforms anywhere), and it seems weird to provide this information in such a form. It would be better if we directly produced a ray -- it's always possible to get a rigid transform out of a ray if desired.

ARCore defines it more rigidly, and the tests follow this definition. Note that the only thing actually dependent on the planes is the intersection point and normal, everything else can be computed given the point, normal, and presupplied input ray.

To me the use of poses/rigid transforms here doesn't make sense, it's fundamentally not a pose. We can attach additional made-up axes to it (the way ARCore does) to make it into a pose, but I don't see the practical use of doing this. The user will likely wish to consume this information in ray form as well, I mostly see users using the position, and if they actually use the orientation it would be to obtain a direction vector.

I suppose using it as a pose can be useful for positioning little indicator objects on the place the hit test actually occurred, but as mentioned before the user has enough information to calculate this for themselves, and this doesn't convince me that an arbitrary choice of X and Z axes is useful.

The status quo is that the tests don't match the API as specced, and that must be fixed one way or the other at the very least.

I see three paths forward for this:

• Turn getPose() into getRay(): IMO this is ideologically the best option, however it's a major breaking change for a somewhat-shipped API and probably practically the worst option.
• More rigidly define the orientation of the hit test result transform: I ... don't like this. This is spending compute cycles on doing math that's not actually going to be used, and we'd have to pick some arbitrary convention.
• Explicitly mention that this transform is not fully defined. Update the tests to be okay with that, i.e. by checking the position and "y axis rotated by orientation" directly.

Thoughts? Is there a purpose for the result being a full pose?

cc @bialpio @toji

(also cc @thetuvix to inform OpenXR hit testing API choices)

[bialpio]

I think it's still better to present the result as a pose, even if the Z and X axes are unconstrained. My goal here was to make it familiar & consistent with frame.getPose(spaceA, spaceB). In addition, poses are very explicit about being expressed relative to reference spaces & carry additional information with them (emulatedPosition), whereas I thought the intent of the XRRay was to be a mathematical concept (similar to XRRigidTransform).

I think we can explicitly specify that X and Z axes are undefined on the returned pose. The WPTs will need to be relaxed in this case, but that should be ok.

[blairmacintyre]

I think the issue is really interesting, and I agree with both of you. 😄

I'd want getPose because that's such a common use and if the Z and X axes are arbitrary, many folks will be just as happy letting the system choose them for them, then doing the obvious math to do it themselves.

Perhaps simply doing what a physics collision system would do, and adding getHitPoint and getNormal (which are the two things you are encapsulating in your getRay) would make sense. There is the collision point, and there is the surface normal at the point of collision, and there is the systems "best guess at a reasonable pose"...

[Manishearth]

Hmm, emulatedPosition is a good point.

I think XRRays are also relative to a space given the way we supply them. I think that while symmetry is valuable, it's not worth it to make the API overall worse to get that symmetry, since now the normal vector is not directly exposed. But this is kind of academic anyway since we can't make this breaking change.

Could you update the relevant tests? We basically need a hit_test_transform_equal function that checks if positions are the same and the orientations transform the y-axis the same way.

[Manishearth]

Perhaps simply doing what a physics collision system would do, and adding getHitPoint and getNormal (which are the two things you are encapsulating in your getRay) would make sense. There is the collision point, and there is the surface normal at the point of collision, and there is the systems "best guess at a reasonable pose"...

I like this, this is similar to the Unity API.

What if we created an XRHitTestPose: XRPose with these APIs (as attributes, not methods)? We continue to allow implementors to arbitrarily pick how X/Z get rotated in the transform field.

The tests can be updated to check these and we can have additional checks that ensure that the transform correctly matches the point/normal.

[bialpio]

I think that the result of a hit test is semantically closer to XRPose (position and orientation relative to some space) than to XRRay (infinite half-line expressed in some space). There is a physical similarity between XRPose with 2 unconstrained axes and XRRay, but to me it seems like not a valid enough reason to use XRRay to represent the results of hit test. XRHitTestPose sounds like something worth considering, but my approach to the APIs was to expose capabilities that are not available - plane normal is already encapsulated in the pose and can be extracted in multiple ways. There is a lot of helpers we could add to spec, I'm just not convinced it's where our current efforts should go to.

[blairmacintyre]

I am not super-happy with XRRay over XRPose because (as you say @bialpio) it's not really "correct" either.

My preference would be to add update HitTestResult with a getHitLocation and getHitNormal (or similar names) to retrieve the collision point and normal, and define the pose relative to them (as discussed above).

I think many people will want the pose, because a common use is to just display some content at the location of the hit, and letting the API choose the undefined two axes is just as good (and simpler, even) than doing it yourself. But, there will be other uses that really want the point and vector, so giving it directly, seems right.

(pardon my crap IDL, I'm sure what I put below isn't exactly right, but you see what I'm saying -- perhaps I should be passing in a space, for example).

//
// Hit Test Results
//
[SecureContext, Exposed=Window]
interface XRHitTestResult {
  XRPose? getPose(XRSpace baseSpace);
  DOMPointReadOnly? getPosition();
  DOMPointReadOnly? getNormal();
};

[Manishearth]

Yeah to be clear I'm past suggesting XRRay, I prefer Blair's idea, though I think it should be on a subclass of XRPose, not on XRHitTestResult, since XRHitTestResult doesn't have a "default space" to relate to.

interface XRHitTestPose: XRPose {
  DOMPointReadOnly? getPosition();
  DOMPointReadOnly? getNormal();
}

[blairmacintyre]

I was suggesting the other way to avoid creating a pose (with "junk" in it) that we don't need.

I admit I was also avoid having to waste "work" computing (and allocating) poses that you don't want or need. By the time you create a pose, the normal is just the Y vector and the position is well defined too. This concern is probably not a big deal (since there won't be piles of hit tests per frame)

[Manishearth]

I was suggesting the other way to avoid creating a pose (with "junk" in it) that we don't need.

Attributes can be lazy evaluated. Implementation wise this would mean that the transform is stored as nullable, and constructed when necessary based on the type.

The thing you proposed can work if getPosition() and getNormal() have an XRSpace argument. In that case I think it makes more sense to me, and i have no strong preference.

---

@bialpio @toji which do you prefer:

interface XRHitTestResult {
  XRPose? getPose(XRSpace baseSpace);
  DOMPointReadOnly? getPosition(XRSpace baseSpace);
  DOMPointReadOnly? getNormal(XRSpace baseSpace);
};

vs

interface XRHitTestResult {
  XRHitTestPose? getPose(XRSpace baseSpace);
}

interface XRHitTestPose : XRPose {
  readonly attribute DOMPointReadOnly position;
  readonly attribute DOMPointReadOnly normal;
};

[bialpio]

I actually prefer another option, which is to refrain from adding helpers until we get strong signals that they are needed. I'm worried that this is a slippery slope and we'll turn this (and the main spec) into a bunch of mathematical utilities. Especially given that it seems to me that both of those methods could be written as one-liners in JS, something along the lines of:

function getPosition(pose) {
  return pose.transform.position;
  // alternatively:
  // return DOMMatrix.fromFloat32Array(pose.transform.matrix).transformPoint(new DOMPoint(0, 0, 0, 1));
}

function getNormal(pose) {
  return DOMMatrix.fromFloat32Array(pose.transform.matrix).transformPoint(new DOMPoint(0, 1, 0, 0));
}

[Manishearth]

I don't consider these helpers: I consider these to be the primary API we should have had in the first place, with .transform being a helper, if anything. We can't get rid of transform now because of backwards compatibility, but it's not the primary way we should be exposing this data to users as they will mostly care about positions and normals.