Skip to content

Commit be60ecd

Browse files
committed
Thumbnails: Add supplementary metadata to getAvailableThumbnailTracks
Based on #1496 Problem ------- We're currently trying to provide a complete[1] and easy to-use API for DASH thumbnail tracks in the RxPlayer. Today the proposal is to have an API called `renderThumbnail`, to which an application would just provide an HTML element and a timestamp, and the RxPlayer would do all that's necessary to fetch the corresponding thumbnail and display it in the corresponding element. The API is like so: ```js rxPlayer.renderThumbnail({ element, time }) .then(() => console.log("The thumbnail is now rendered in the element")); ``` This works and seems to me very simple to understand. Yet, we've known of advanced use cases where an application might not just want to display a single thumbnail for a single position. For example, there's very known examples where an application displays a window of multiple thumbnails at once on the player's UI to facilitate navigation inside the content. To do that under the solution proposed in #1496, an application could just call `renderThumbnail` with several `element` and `time` values. Yet for this type of feature, what the interface would want is not really to indicate a `time` values, it actually wants basically a list of distinct thumbnails around/before/after a given position. By just being able to set a `time` value, an application is blind on which `time` value is going to lead to a different timestamp (i.e. is the thumbnail for the `time` `11` different than the thumbnail for the `time` `12`? Nobody - but the RxPlayer - knows). So we have to find a solution for this [1] By complete, I here mean that we want to be able to handle its complexities inside the RxPlayer, to ensure complex DASH situations like multi-CDN, retry settings for requests and so on while still allowing all potential use cases for an application. Solution -------- In this solution, I experiment with a second thumbnail API, `getAvailableThumbnailTracks` (it already exists in #1496, but its role there was only to list the various thumbnail qualities, if there are several size for example). As this solution build upon yet stays compatible to #1496, I chose to open this second PR on top of that previous one. I profit from the fact that most standardized thumbnail implementations I know of (BIF, DASH) seem follow the principle of having evenly-spaced (in terms of time) thumbnails (though I do see a possibility for that to change, e.g. to have thumbnails corresponding to "important" scenes instead, so our implementation has to be resilient). So here, what this commit does is to add the following properties (all optional) to a track returned by the `getAvailableThumbnailTracks` API: - `start`: The initial `time` the first thumbnail of that track will apply to - `end`: The last `time` the last thumbnail of that track will apply to - thumbnailsPerSegment: Individual thumbnails may be technically part of "segments" containing multiple consecutive thumbnails each. `thumbnailsPerSegment` is the number of thumbnails each of those segments contain. For example you could have stored on the server a segment which is a grid of 2 x 3 (2 horizontal rows and * 3 vertical columns) thumbnails, which the RxPlayer will load at once then "cut" the right way when calling `renderThumbnail`. In that example, `thumbnailsPerSegment` would be set to `6` (2*3). Note that the last segment of a content may contain less thumbnails as anounced here depending on the duration of the content. - `segmentDuration`: The "duration" (in seconds) each segments of thumbnails applies to (with the exception of the last thumbnail, which just fills until `end`) Then, an application should have all information needed to calculate a `time` which correspond to a different thumbnail. Though this solution lead to a minor issue: by letting application make the `time` operation themselves with `start`, `end`, `segmentDuration` and so on, there's a risk of rounding errors leading to a `time` which does not correspond to the thumbnail wanted but the one before or after. To me, we could just indicate in our API documentation to application developers that they should be extra careful and may add an epsilon (or even choose a `time` in the "middle" of thumbnails each time) if they want that type of thumbnail list feature. Thoughts?
1 parent 2275810 commit be60ecd

File tree

2 files changed

+57
-0
lines changed

2 files changed

+57
-0
lines changed

src/main_thread/api/public_api.ts

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -856,6 +856,10 @@ class Player extends EventEmitter<IPublicAPIEvent> {
856856
width: Math.floor(t.width / t.horizontalTiles),
857857
height: Math.floor(t.height / t.verticalTiles),
858858
mimeType: t.mimeType,
859+
start: t.start,
860+
end: t.end,
861+
thumbnailDuration: t.tileDuration,
862+
thumbnailsPerSegment: t.horizontalTiles * t.verticalTiles,
859863
};
860864
});
861865
}

src/public_types.ts

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1395,6 +1395,59 @@ export interface IThumbnailTrackInfo {
13951395
* `image/jpeg` or `image/png`.
13961396
*/
13971397
mimeType: string | undefined;
1398+
/**
1399+
* Starting `position` the first thumbnail of this thumbnail track applies to,
1400+
* if known.
1401+
*/
1402+
start: number | undefined;
1403+
/**
1404+
* Ending `position` the last thumbnail of this thumbnail track applies to,
1405+
* if known.
1406+
*/
1407+
end: number | undefined;
1408+
/**
1409+
* Individual thumbnails may be technically part of "segments" containing
1410+
* multiple consecutive thumbnails each.
1411+
*
1412+
* `thumbnailsPerSegment` is the number of `thumbnails` each segments have.
1413+
*
1414+
* For example you could have stored on the server a segment which is a grid
1415+
* of 2 x 3 (2 horizontal rows and 3 vertical columns) thumbnails, which the
1416+
* RxPlayer will load at once then "cut" the right way when calling
1417+
* `renderThumbnail`. In that example, `thumbnailsPerSegment` would be set to
1418+
* `6` (2*3).
1419+
*
1420+
* Note that the last segment of a content may contain less thumbnails as
1421+
* anounced here depending on the duration of the content.
1422+
*
1423+
* You may want to rely on this information alongside `thumbnailDuration` to
1424+
* construct a list of available thumbnails and/or of available segments of
1425+
* thumbnails.
1426+
*/
1427+
thumbnailsPerSegment: number | undefined;
1428+
/**
1429+
* When loaded, thumbnails are part of so-called "segments" which may contain
1430+
* either a single thumbnail or a grid of them (@see `thumbnailsPerSegment`).
1431+
*
1432+
* This `thumbnail` property indicates a duration in seconds each thumbnail
1433+
* apply to.
1434+
* You can multiply that value with `thumbnailsPerSegment` to get the amount
1435+
* of time each segment applies to.
1436+
*
1437+
* Set to `undefined` either the duration is unknown or if the duration
1438+
* depends from segment to segments.
1439+
*
1440+
* For example, with a `start` set to `10`, an `end` set to `26`, a
1441+
* `thumbnailsPerSegment` set to `2` and a `tileDuration` set to
1442+
* `3`, there should be 3 segments, each with 2 thumbnails of 3 seconds each:
1443+
* 1. A segment of 2 thumbnails for the seconds: 10-16
1444+
* (The first thumbnail in that segment for 10-13, the second for 13-16)
1445+
* 2. A segment of 2 thumbnails for the seconds: 16-22
1446+
* (The first thumbnail in that segment for 16-19, the second for 19-22)
1447+
* 3. A segment of 2 thumbnails for the seconds: 22-26 (the end)
1448+
* (The first thumbnail in that segment for 22-25, the second for 25-26)
1449+
*/
1450+
thumbnailDuration: number | undefined;
13981451
}
13991452

14001453
/**

0 commit comments

Comments
 (0)