Using align with software device in C#

[jb455]

---

Required Info
Camera Model	D400
Firmware Version	5.11.6.250
Operating System & Version	WIn10
Platform	PC
SDK Version	2.24.0 }
Language	C#

Hi,
I had working with SDK 2.14 where I was using a software device to align streams from two different cameras. I have recently updated to the latest SDK (2.24) and now when I align, the depth stream is almost blank (all zeroes apart from the first pixel). Has there been a regression or is there something else I now need to do to get it working? Attached is modified Window.xaml.cs from cs-tutorial-4-software-dev to reproduce the issue.
Thanks
James
Window.xaml.cs.txt

[dorodnic]

Hi @jb455
Thanks for the feedback, we will look into it.
Aside, is it really necessary to use software device for this purpose? Looking back at your contribution from earlier this year (thank you for contributing! :) ) I'd say you could just RegisterExtrinsics between the two depth Stream Profiles and this should be sufficient for Align. Am I wrong?

[jb455]

Hi, thanks for your reply.
I'm actually aligning a depth stream with an external colour stream. I did try just making a software sensor for the colour stream and registering the extrinsics to the actual depth stream without using a software device (and repackaging the real depth stream into a software depth stream) but I couldn't get it working.
I think the frameset passed to align needs to contain both the 'to' and 'from' frames so that align knows the intrinsics of both streams (looking at align.cpp#264, that appears to be the case), and I couldn't find a way to package the depth and colour frames in a single frameset without using a software device.

Ideally for my purposes I would pass align a depth frame, extrinsics to the colour stream, and intrinsics of the colour stream - that would save a lot of overhead with software devices etc.
If I were to implement that, what would be the best way of going about it? Creating a new align constructor which takes in the intrinsics and extrinsics instead of a stream type and maybe setting a bool to have alternative paths throughout wherever it's using to_profile; or an alternative process_frame method?
If the former, I don't quite understand how the current constructor works. The public align(rs2_stream to_stream) redirects to align(to_stream, "Align") at align.cpp#64, which in turn redirects to generic_processing_block(name) at align.h#22, and I don't see how that links back to align.

[hopaa]

Hi to @jb455 and @dorodnic
Any information about the bug? Has it been resolved?
Currently I'm dealing with the same issue. Has it been fixed in the latest released SDK (2.25.0)?

Thanks in advance

[dorodnic]

I still need to look into it, sorry for the delay

_{Sent with GitHawk}

[jb455]

Hi @dorodnic, any news on this?
Thanks!

[jb455]

@dorodnic: we've noticed that the latest firmware (5.11.11.100) isn't compatible with the version of the SDK we're using - it fails when trying to change the visual preset. I'm guessing this is because of the new camera parameter added; depending on where the preset data is stored (firmware or SDK), either when the old SDK sees the new parameter in the JSON it doesn't know what to do with it, or the firmware gets old JSON and is expecting the new parameter. However, we can't update the SDK to mitigate that problem due to the issue in this thread. Is there an interim version of the SDK we can use before alignment was broken but after the new parameters were added?

[jb455]

@dorodnic: I've found the original issue. The problem is that when align tries to get the depth units at https://github.com/IntelRealSense/librealsense/blob/master/src/proc/align.cpp#L259 it instead queries the stereo baseline here for some reason. If you haven't set the baseline as a readonly option, it fails and returns 0 which is why the depth image was blank. To get around that you can use
softDepthSensor.AddReadOnlyOption(Option.StereoBaseline, 1/depthUnits);
which now gives a depth image with nonzero values scaled correctly.

However, I'm now having an issue where the image isn't actually aligning properly; it seems the extrinsics are lost at some point and reset to the identity transformation. I use
softDepthStream.RegisterExtrinsicsTo(softColourStream, extrinsics);
to set the extrinsics, then later calling
var ext = softDepthStream.GetExtrinsicsTo(softColourStream);
returns them correctly, but it doesn't appear as though Align is using them properly.

Should I be setting the stereo baseline to something else maybe? Setting it to the actual stereo baseline of the physical camera doesn't seem to help.

Edit:
Actually, setting baseline to depthUnits (not the inverse) makes it work. I'm fairly sure I tried that a couple of days ago and it wasn't working but there we go!

[RealSenseSupport]

Please kindly check if the problem persists in the latest SDK 2.49 release. Thank you.
https://github.com/IntelRealSense/librealsense/wiki/Release-Notes#release-2490

[RealSenseSupport]

We will close this ticket for now. Please let us know or open a new issue if there is any other problem.
Thank you.