[Proposal]: Thumbnails: Add supplementary metadata to getAvailableThumbnailTracks

[peaBerberian]

EDIT: I added lastThumbnailTime which is an estimate of the start time of the last available thumbnail for a track. This is to help cases where the application wants to load multiple thumbnails until the last one.

Problem

We're currently trying to provide a complete[1] and easy to-use API for DASH thumbnail tracks in the RxPlayer.

Today the proposal is to have an API called renderThumbnail, to which an application would just provide an HTML element and a timestamp, and the RxPlayer would do all that's necessary to fetch the corresponding thumbnail and display it in the corresponding element.

The API is like so:

rxPlayer.renderThumbnail({ element, time })
  .then(() => console.log("The thumbnail is now rendered in the element"));

This works and seems to me very simple to understand.

Yet, we've known of advanced use cases where an application might not just want to display a single thumbnail for a single position. For example, there's very known examples where an application displays a window of multiple thumbnails at once on the player's UI to facilitate navigation inside the content.

Screenshot: this is the interface of a very popular web media player (the official one from the platform for which you installed newpipe instead) where multiple thumbnails are shown at once, under the seek bar. Interestingly, the video is also replaced by a thumbnail in that mode here, I guess to provide a smoother experience when rapidly navigating in the content.

To do that under the solution proposed in #1496, an application could just call renderThumbnail with several element and time values.
Yet for this type of feature, what the interface would want is not really to indicate time values, it actually wants basically a list of distinct thumbnails around/before/after a given position.

By just being able to set a time value, an application is blind on which time value is going to lead to a different thumbnail (i.e. is the thumbnail for the time 11 different than the thumbnail for the time 12? Nobody - but the RxPlayer - knows).

So we have to find a solution for this

[1] By complete, I here mean that we want to be able to handle its complexities inside the RxPlayer to ensure advanced DASH configurations like multi-CDN, retry settings for requests etc., while still allowing all potential use cases for an application.

Solution

In this solution, I experiment with a second thumbnail API, getAvailableThumbnailTracks (it already exists in #1496, but its role there was only to list the various thumbnail qualities, if there are several size for example). As this solution build upon yet stays compatible to #1496, I chose to open this second PR on top of that previous one.

I profit from the fact that most standardized thumbnail implementations I know of (BIF, DASH) seem to follow the principle of having evenly-spaced (in terms of time) thumbnails (though I do see a possibility for that to change, e.g. to have thumbnails corresponding to "important" scenes instead, so our implementation has to be resilient).

So here, what this PR does is to add the following properties (all optional) to a track returned by the getAvailableThumbnailTracks API:

• start: The initial time the first thumbnail of that track will apply to

• end: The last time the last thumbnail of that track will apply to.

  ⚠️ For live contents (which I guess should be the main type of contents relying on this, BIF thumbnails are good enough for VoD contents), for now that end will announce the future expected end of the Period if already known (for example thanks to a Period@end or a SegmentTemplate@endNumber attribute) - and undefined if unknown.
  In this current configuration, an application has to also request another API, like getLivePosition to know the maximum position it can currently load thumbnails from

• thumbnailsPerSegment: Individual thumbnails may be technically part of "segments" containing multiple consecutive thumbnails each.
  thumbnailsPerSegment is the number of thumbnails each of those segments contain.

  For example you could have stored on the server a segment which is a grid of 2 x 3 (2 horizontal rows and * 3 vertical columns) thumbnails, which the RxPlayer will load at once then "cut" the right way when calling renderThumbnail. In that example, thumbnailsPerSegment would be set to 6 (2*3).

  Note that the last segment of a content may contain less thumbnails as announced here depending on the duration of the content.

• thumbnailDuration: The "duration" (in seconds) each thumbnail applies to

• lastThumbnailTime: The last starting time for the last thumbnail currently available in that track. For live contents this value may become outdated quickly, but it gives an indication on the last thumbnail currently served.

Then, an application should have all information needed to calculate a time which correspond to a different thumbnail.

Though this solution lead to a minor issue: by letting application make the time operation themselves with start, end, thumbnailDuration and so on, there's a risk of rounding errors leading to a time which does not correspond to the thumbnail wanted but the one before or after. To me, we could just indicate in our API documentation to application developers that they should be extra careful and may add an epsilon (or even choose a time in the "middle" of thumbnails each time) if they want that type of thumbnail list feature.

Thoughts?

[peaBerberian]

I'm still not sure with the approach. Let's say an app wants to show 7 thumbnails on the bottom at once with the current ""playing thumbnail"" being at the center, like in a carousel (e.g. though with less images at once:
)

To calculate the list of thumbnails to display, they would now do:

const IMAGES_BEFORE = 3; 
const IMAGES_AFTER = 3;

const thumbnailTrack = player.getAvailableThumbnailTracks()?.[0];

function getThumbnailCarrousel(thumbnailTrack) {
  const position = player.getPosition();

  // behind, after, and the current thumbnail
  const arrOfThumbnails = new Array(IMAGES_BEFORE + IMAGES_AFTER + 1);

  const leftImagesToShow = IMAGES_BEFORE + IMAGES_AFTER + 1;
  const relativeCurrentPosition = position - thumbnailTrack.start;
  const relativeCurrentThumbnailStart = relativeThumbnailTime - (relativeThumbnailTime % thumbnailTrack.thumbnailDuration);

  arrOfThumbnails[IMAGES_BEFORE + 1] = relativeCurrentThumbnailStart + thumbnailTrack.start;
  leftImagesToShow--; // We know for now only the current thumbnail time
  
  const thumbnailIdx = Math.round(relativeCurrentThumbnailStart / thumbnailTrack.thumbnailDuration)

  if (thumbnailIdx < IMAGES_BEFORE) {
    leftImagesToShow -= thumbnailIdx;
    for (let i = 0; i < thumbnailIdx; i++) {
      arrOfThumbnails[i] = thumbnailTrack.start + thumbnailTrack.thumbnailDuration * i;
    }
  } else {
    // TODO: check if we have less IMAGES than IMAGES_AFTER and if we do add some more images before

    leftImagesToShow -= IMAGES_BEFORE;
    for (let i = 0; i < IMAGES_BEFORE; i++) {
      arrOfThumbnails[i] = thumbnailTrack.start + thumbnailTrack.thumbnailDuration * i;
    }
  }
  
  // TODO: Same thing with IMAGES_AFTER
  // ....

  // Remove empty array slots/undefined
  return arrOfThumbnails.filter(a => typeof a === "number");
}

// Then something like
if (thumbnailTrack) {    
  const list = getThumbnailCarrousel(thumbnailTrack);
  for (const l of list) {
    renderThumbnail({
      container: someElement,
      thumbnailTrackId: thumbnailTrack.id,
      time: l,
    });
  }
}

I only did a small fraction of the code but it already looks very awkward and hard. Not even taking into consideration possible rounding errors leading to the wrong thumbnail.

---

Maybe giving a simple iterative thumbnail number (from 0, to 1, to 2 etc.) would be easier to handle for application?

We could e.g. add the following metadata to a thumbnail track:

1. getThumbnailNumber function taking a position in seconds, in which in the RxPlayer would do something like:

   function getThumbnailNumber(position) {
     const relativeCurrentPosition = position - thumbnailTrack.start;
     return Math.floor(relativeCurrentPosition) / thumbnailTrack.thumbnailDuration;
   }  

2. a lastThumbnailNumber which returns the current last thumbnail "number" for the thumbnail track.

And the RxPlayer could also take the number in renderThumbnail, e.g.:

rxPlayer.renderThumbnail({
  container,
  thumbnailTrackId,
  periodId,
  number, // Here
})

Then the application can directly:

const IMAGES_BEFORE = 3; 
const IMAGES_AFTER = 3;

const thumbnailTrack = player.getAvailableThumbnailTracks()?.[0];

function getThumbnailCarrousel(thumbnailTrack) {
  const position = player.getPosition();

   // behind, after, and the current thumbnail
  const arrOfThumbnailNumbers = new Array(IMAGES_BEFORE + IMAGES_AFTER + 1);

  const currentThumbnailIndex = thumbnailTrack.getThumbnailNumber(position);
  arrOfThumbnailNumbers[IMAGES_BEFORE + 1] = currentThumbnailIndex;
  leftImagesToShow--; // We know for now only the current thumbnail time

  if (currentThumbnailIndex < IMAGES_BEFORE) {
    leftImagesToShow -= currentThumbnailIndex;
    for (let i = 0; i < currentThumbnailIndex; i++) {
      arrOfThumbnailNumbers[i] = i;
    }
  } else {
    // TODO: check if we have less IMAGES than IMAGES_AFTER and if we do add some more images before

    leftImagesToShow -= IMAGES_BEFORE;
    for (let i = 0; i < IMAGES_BEFORE; i++) {
      arrOfThumbnailNumbers[i] = i;
    }
  }
  
  // TODO: Same thing with IMAGES_AFTER
  // ....

  // Remove empty array slots/undefined
  return arrOfThumbnails.filter(a => typeof a === "number");
}

if (thumbnailTrack) {    
  const list = getThumbnailCarrousel(thumbnailTrack);
  for (const l of list) {
    renderThumbnail({
      container: someElement,
      thumbnailTrackId: thumbnailTrack.id,
      periodId: player.getCurrentPeriod()?.id,
      number: l,
    });
  }
}

Much less awkward no?

[github-actions]

✅ Automated performance checks have passed on commit 871318818c876d1829bc1c45344fe4b3c20c022f with the base branch dev.

Details

Performance tests 1st run output

No significative change in performance for tests:

Name	Mean	Median
loading	19.36ms -> 19.48ms (-0.118ms, z: 0.63252)	27.15ms -> 27.00ms
seeking	20.38ms -> 19.03ms (1.351ms, z: 0.45186)	10.65ms -> 10.80ms
audio-track-reload	25.44ms -> 25.47ms (-0.028ms, z: 0.88700)	37.35ms -> 37.20ms
cold loading multithread	45.01ms -> 44.21ms (0.804ms, z: 15.56768)	66.45ms -> 65.25ms
seeking multithread	72.69ms -> 82.69ms (-10.002ms, z: 0.70204)	9.45ms -> 9.45ms
audio-track-reload multithread	25.20ms -> 25.05ms (0.147ms, z: 1.18344)	37.05ms -> 36.90ms
hot loading multithread	14.55ms -> 14.37ms (0.180ms, z: 6.09337)	21.30ms -> 21.00ms

[github-actions]

✅ Automated performance checks have passed on commit c00cb980b26d9a7ff0e63fdcd681c1cde020458b with the base branch dev.

Details

Performance tests 1st run output

No significative change in performance for tests:

Name	Mean	Median
loading	21.62ms -> 22.33ms (-0.711ms, z: 0.64213)	30.00ms -> 30.15ms
seeking	14.04ms -> 20.05ms (-6.006ms, z: 0.01187)	11.25ms -> 11.10ms
audio-track-reload	27.84ms -> 27.95ms (-0.103ms, z: 0.22529)	40.80ms -> 40.80ms
cold loading multithread	49.35ms -> 48.30ms (1.050ms, z: 15.11778)	72.60ms -> 71.25ms
seeking multithread	35.27ms -> 37.77ms (-2.502ms, z: 0.71987)	10.20ms -> 10.20ms
audio-track-reload multithread	26.77ms -> 26.68ms (0.085ms, z: 1.62586)	39.35ms -> 39.30ms
hot loading multithread	16.80ms -> 16.79ms (0.018ms, z: 2.05356)	24.60ms -> 24.45ms