Add backchannel support for RTSP server

[seydx]

Currently, there is no way for clients to send audio data to go2rtc via RTSP backchannel, as the RTSP server lacks a backchannel implementation.

This PR adds a backchannel to the stream (when backchannel=1) and thus allows clients to send audio data to go2rtc.

Tested streams so far:

• webrtc:ws://
• tapo://
• rtsp:// with backchannel
• onvif://
• ring:

I think other streams with backchannel should also work

[felipecrs]

@seydx, out of curiosity, what use case this enables? Connecting a go2rtc server to another?

[seydx]

@seydx, out of curiosity, what use case this enables? Connecting a go2rtc server to another?

This implementation allows me to send audio data directly to go2rtc using my RTSP client. The data is sent interleaved over the connected socket

Since go2rtc offers two-way audio for some cameras, e.g. tapo etc, the rtsp client can connect to the go2rtc rtsp link of the tapo camera and send audio data directly

This functionality is natively available in WebRTC, where you can send audio directly from the browser to the camera. With this implementation, virtually anyone can do it, regardless of the programming language​​​​​​​​​​​​​​​​ and environment, since rtsp ist just a tcp connection

[SVilgelm]

This is great, I would really like to this feature, hopefully it will allow me to have two-way audio with my kasa cameras via scrypted.

I'm using go2rtc to capture the videos and transcode audio from kasa cameras, then i connect the cameras to scrypted via rtsp to have motion detection and then to HomeKit :)

[SVilgelm]

@AlexxIT could you please review this PR?

[AlexxIT]

It's marked as assigned, which means it's in the plans. But now I have over 100 of these markings.
https://github.com/AlexxIT/go2rtc/issues?q=is%3Aopen+assignee%3AAlexxIT

[seydx]

It's marked as assigned, which means it's in the plans. But now I have over 100 of these markings. https://github.com/AlexxIT/go2rtc/issues?q=is%3Aopen+assignee%3AAlexxIT

Hey Alex, I just wanted to let you know that I've had it in camera.ui for months now and have been testing it extensively, and so far no problems have come up.

[AlexxIT]

Thanks!

[felipecrs]

Does it mean that clients may no longer need to use go2rtc-specific APIs to support two-way audio?

If that's correct, maybe that also means we no longer need to expose go2rtc API to outside in certain cases, and WebRTC port would be enough.

[seydx]

Does it mean that clients may no longer need to use go2rtc-specific APIs to support two-way audio?

If that's correct, maybe that also means we no longer need to expose go2rtc API to outside in certain cases, and WebRTC port would be enough.

Why would you expose go2rtc api to outside (except webrtc/rtsp), for two-way audio ?

[felipecrs]

Advanced Camera Card (formerly Frigate Card) needs it.

AFAIK even @AlexxIT's WebRTC card needs it, but the integration part of it is responsible for proxying go2rtc API through Home Assistant (I think, not completely sure).

Even the Frigate integration proxies go2rtc API for the Frigate Card, but that only works for go2rtc that runs within Frigate itself.

[seydx]

Advanced Camera Card (formerly Frigate Card) needs it.

AFAIK even @AlexxIT's WebRTC card needs it, but the integration part of it is responsible for proxying go2rtc API through Home Assistant (I think, not completely sure).

Even the Frigate integration proxies go2rtc API for the Frigate Card, but that only works for go2rtc that runs within Frigate itself.

Ah ok, now I understand what you mean. I'm not very familiar with Frigate.

In that case, the go2rtc API is mainly needed for the Websocket connection because of WebRTC signaling and MSE/MP4 etc. since these binary data come over Websocket.

In the case of WebRTC, the API is only necessary for signaling, since a direct connection is established afterwards.

I think I'm doing something similar to Frigate, using my app as a proxy to communicate with go2rtc. It's sufficient if go2rtc is only accessible via localhost, since in that case my app serves as a proxy.

The 2-way audio for RTSP implemented here is not relevant in this case, since it only works with a direct RTSP connection to the camera. This is because the 2-way audio RTP data going from the client to the camera are interleaved, meaning they're sent over the same TCP connection where the video/audio data from the camera come from.

[felipecrs]

Got it. Thank you.

And yeah, even in Frigate own interface/web-ui, 2-way audio and live streaming works thanks to the go2rtc API being proxied through the Frigate web port.

go2rtc port doesn't need to be directly exposed, except in some scenarios like using the Advanced Camera Card with go2rtc but without Frigate integration.

[AlexxIT]

This PR adds backchannel support to the go2rtc RTSP server. So any software that supports it can get two way audio via RTSP from go2rtc.

[AlexxIT]

https://github.com/AlexxIT/go2rtc/releases/tag/v1.9.9