[Proposal] Improve FREEZING mechanisms and add Representation avoidance mechanism

[peaBerberian]

Situation

We have lately seen on some LG and Philips TVs what we call "infinite FREEZING" occurences: the playback position (the HTMLMediaElement's currentTime property) was not advancing and most of the time video media was not playing either (though in some occurences audio was) despite having largely enough media data in media buffers. Sadly our usual trick to restart playback, seeking close to the current position, didn't seem to have an effect.

Most of those freezing cases were happening as playback switched from a video quality to another (sometimes both had to be specific, sometimes only the destination one had an impact).

It is probably better to detect which situations causes which problems on which devices to work-around it either in the RxPlayer (when for example we KNOW that a device has issues with a given codec in general) or inside the application (when we only reproduced it on a specific application and there's many unknowns left). Yet we found out after encountering issues again and again, especially on some smart TVs, that it may not be realistic to catch in advance (before production) all potential breakages that could exist.

So instead, we thought about the possibility to define an heuristic detecting that we're probably encountering this type of issue, to then work-around it automatically.

Rough idea of the solution

So, the idea is to detect when a FREEZING case occurs right when switching from a given video/audio quality to another. When that's the case:

1. We first try to "un-freeze" playback, by e.g. performing a small seek. This is what we already do today in FREEZING cases.

2. If that doesn't work (we're still freezing), I added here a second mechanism: we "avoid" the corresponding new quality played, so it isn't played anymore in most cases.

So basically once we see that switching to another quality lead to FREEZING, we avoid playing that quality for the rest of playback.

Unlike other scenarios where we prevent playback of qualities (in the RxPlayer called Representation), here the "avoidance mechanism" is only considered if there's other, non-avoided, qualities we can play on the current track: meaning that if all available qualities are "avoided", we will play them again. This is to protect against errors in cases where all qualities would be avoided: because our heuristic is not 100% sure science, we would here prefer there to keep playback potentially happening (the alternative would have been stopping playback on an error).

As of now, the "Representation avoidance" has no impact on the API: corresponding Representation are still returned as if they are playable through our getVideoTrack API for example, and the user can still choose to call lockVideoRepresentation on avoided Representation without knowing. As of now, the "main thread" part of the code isn't even aware of such avoidance happening, for it we're just curiously never seem to be playing some Representation. This also means that our DEBUG_ELEMENT feature won't even notify for now if there's avoidance happening.

This may change in the future, but for now, this is because I did not want to complexify this experiment too much.

PS: I like this "avoidance" concept, as it seems to me to be portable for much more future cases where we would prefer not playing qualities because they seem to present issues (e.g. if MEDIA_ERR_DECODE or BUFFER_APPEND_ERROR errors seem to only happen with some Representation).

Bonus: also reloading in other cases

There are several other posibilities for infinite FREEZING: Period changes gone wrong, random decoding issues, etc.

To also provide a better solution for those cases, I choose to "reload" (in the RxPlayer, "reloading" means removing and re-creating all the media buffers, leading to a temporary black screen) if the initial "un-freezing" attempt (the seek-based one) didn't have any effect.

Unlike when the freezing seems to be linked to a quality change though, there's no Representation avoidance happening in that case.

Implementation

FreezeResolver

I renamed the very-specific DecipherabilityFreezeDetector class - which only handled freezes linked to DRM - into a more general FreezeResolver class, which will now perform all un-freezing attempts (even the seekinf one, which previously happened in the RebufferingController in main thread whereas it's now in core - thus optionally running in our Worker).

The FreezeResolver's onObservation method has to be called even when playback goes well, at each produced media observation, because it needs to construct its history of played segments (see next chapter).

Quality switch detection

Knowing whether we switched from a given media quality to another is not straightforward and we can never really know for sure as there's edge cases where it might be device-specific (e.g. segment replacements, segment pushing still pending etc.).

However, we have a rough idea by inspecting the HTMLMediaElement's currentTime property (which is roughly the current media position being played), and the content of our SegmentInventory class, which stores metadata about all media segments available in the buffer.

From there, I maintain a short-term history in the FreezeResolver of what seemed to be the current video AND audio quality played. If a FREEZING, which we do not seem to be able to fix by seeking, seems to coincide with a quality switch, we propose to avoid the Representation.

For now this also lead to a RELOADING operation, though it may not even be required in some cases (e.g. replacing segments and seeking in place might do the trick). Yet reloading should have more chance of fixing it (even though it leads to a temporary black screen, where a still video frame would have been less disruptive for users).

Avoiding Representation

Then, an AdaptationStream is able to filter out avoided Representation (unless it has no choice), when asking the AdaptiveRepresentationSelector (managing our Adaptive BitRate logic) which Representation should be played (though I'm still not sure whether the AdaptiveRepresentationSelector should actually be the one doing that?).

Where we go from there

This heuristic seems somewhat risky. as we're essentially blacklisting qualities from ever be playing again on the current content.

So what I thought was to put the avoidance mechanism behind an experimental experimental.enableRepresentationAvoidance loadVideo option, that applications would have to enable (making it also possible to disable at any time easily through some config).

However the reload mechanism if un-freezing fails seems OK to me for enabling it by default.

If we do notice clear improvements, we might think of enabling the avoidance mechanism by default, removing the
experimental.enableRepresentationAvoidance option.

Thoughts?

[Florent-Bouisset]

While I understand "deprecated" representation, it feels like it's mean that it is no longer supported because the representation uses aged codecs or DRM encrypting.
I could see in the future an usage of marking as deprecated H.264 codecs and prefer H.265 one; or even mark a content encrypted with CENC as deprecated to prefer CBCS encrypted one.

What do you think about "Unstable" or "Avoided" ?

[peaBerberian]

it feels like it's mean that it is no longer supported because the representation uses aged codecs or DRM encrypting

I don't see deprecation as a lack of support but more of a discouragement on its usage personally, but I see your point.

I'll check with Avoidance/Avoided/Avoid if it makes sense in the various usages.
We would thus have something like a experimental.enableRepresentationAvoidance option

[peaBerberian]

Done (this late, because during very uneventful on-call duties ;p)

I relied on the "Avoidance" name for now, as "unstable" could mean that the issue is with the media itself - yet we don't know that, all we know is that the device seems to have issues decoding it.

[github-actions]

Automated performance checks have been performed on commit edfece2d44bfc395d43af6fe2f16d045cf38cf96 with the base branch dev.

Tests results

✅ Tests have passed.

Performance tests 1st run output

No significative change in performance for tests:

Name	Mean	Median
loading	19.14ms -> 19.03ms (0.115ms, z: 1.01724)	26.25ms -> 26.25ms
seeking	11.81ms -> 14.43ms (-2.626ms, z: 1.00360)	11.10ms -> 11.10ms
audio-track-reload	25.33ms -> 25.26ms (0.066ms, z: 1.37644)	37.20ms -> 36.95ms
cold loading multithread	47.11ms -> 46.69ms (0.416ms, z: 7.57125)	69.30ms -> 68.25ms
seeking multithread	18.46ms -> 17.18ms (1.275ms, z: 1.05940)	10.35ms -> 10.35ms
audio-track-reload multithread	25.44ms -> 25.23ms (0.210ms, z: 1.98160)	37.35ms -> 37.05ms
hot loading multithread	15.05ms -> 14.82ms (0.227ms, z: 4.66057)	21.60ms -> 21.45ms

If you want to skip performance checks for latter commits, add the skip-performance-checks label to this Pull Request.

[github-actions]

Automated performance checks have been performed on commit abd2d7b6405bbe41cfe5145b2af73b5564516aa1 with the base branch dev.

Tests results

✅ Tests have passed.

Performance tests 1st run output

No significative change in performance for tests:

Name	Mean	Median
loading	19.45ms -> 19.53ms (-0.081ms, z: 1.71984)	26.55ms -> 26.70ms
seeking	15.99ms -> 11.40ms (4.588ms, z: 1.97947)	11.25ms -> 11.25ms
audio-track-reload	26.37ms -> 26.43ms (-0.058ms, z: 0.58161)	38.40ms -> 38.55ms
cold loading multithread	48.09ms -> 47.91ms (0.185ms, z: 8.58268)	70.20ms -> 69.30ms
seeking multithread	15.26ms -> 20.61ms (-5.348ms, z: 1.33841)	10.35ms -> 10.50ms
audio-track-reload multithread	26.30ms -> 26.13ms (0.170ms, z: 1.98670)	38.70ms -> 38.55ms
hot loading multithread	16.40ms -> 16.29ms (0.103ms, z: 2.23865)	23.85ms -> 23.70ms

If you want to skip performance checks for latter commits, add the skip-performance-checks label to this Pull Request.