Enable VideoStream H264 on the web#10189
Conversation
…y to new struct, reshuffle some symbols
…er use video encoding details directly
… got newly discovered
Wumpf
added
🕸️ web
|
Web viewer built successfully. If applicable, you should also test it:
Note: This comment is updated whenever you push a commit. |
| // TODO(#10184): Add support for AV1. | ||
| //AV1, | ||
|
|
||
| /// VP8 |
There was a problem hiding this comment.
don't think it makes sense to support VP8, which is why it didn't get a ticket
| on_output: Arc<Mutex<Option<Arc<OutputCallback>>>>, | ||
| } | ||
|
|
||
| fn try_get_sps_from_avcc(avcc: &re_mp4::Avc1Box, debug_name: &str) -> Option<sps::SeqParameterSet> { |
There was a problem hiding this comment.
no need to look at the avcc in detail here anymore - we do so now agnostically by having a look at the new VideoEncodingDetails
| /// The sample is the start of a GOP and encoding details have been extracted. | ||
| StartOfGop(VideoEncodingDetails), |
There was a problem hiding this comment.
in theory we can also detect a gop start but fail to extract those details. Went there at first but got needlessly complex for what effectively has to be a broken gop
| if reader.nal_handler_ref().detected_gop() { | ||
| return true; | ||
| // In case of SPS parsing failure keep going. | ||
| // It's unlikely, but maybe there's another SPS in the chunk that succeeds parsing. |
There was a problem hiding this comment.
... tbh I don't even now what that means if that would happen. But with everything video I generously assume everything can happen...
| let result = detect_h264_annexb_gop(sample_data); | ||
| assert_eq!( | ||
| result, | ||
| Ok(GopStartDetection::StartOfGop(VideoEncodingDetails { |
There was a problem hiding this comment.
the fact that this above ai generated bytes actually generated something we can parse makes me really curious about that IDR frame in there 😆
| Yuv420, | ||
|
|
||
| /// No subsampling at all, since the format is monochrome. | ||
| Monochrome, |
There was a problem hiding this comment.
merging monochrome into the ChromaSubsamplingModes made a bunch more things saner!
| pub coded_dimensions: Option<[u16; 2]>, | ||
| /// Various information about how the video was encoded. | ||
| /// | ||
| /// Should any of this change during the lifetime of a decoder, it has to be reset. |
There was a problem hiding this comment.
another piece of loose contract on the mutability of VideoDataDescription. Makes me rather uneasy, but it's workable
| /// Returns the subsampling mode of the video. | ||
| /// | ||
| /// Returns None if not detected or unknown. | ||
| pub fn subsampling_mode(&self) -> Option<ChromaSubsamplingModes> { |
There was a problem hiding this comment.
this parsing code moved into mp4.rs where it belongs
2 participants
Related
What
In order to enable streaming on the web we have to extract a fully qualified codec descriptor string, as specified by the webcodecs codec registry
Doing so means that we have to even deeper inspect the video bistream and extract this information on-the-fly.
Setting the stage:
is_sync) is required to contain an IDR frame AND an Sequence Parameter Set (SPS)Therefore, this PR:
VideoDataDescriptionto not be as little reliant on AVCC, splitting out the optionalVideoEncodingDetails(Q: better name? it's not like we're encoding something, it's more like some loosy goose information on what got encoded how)VideoEncodingDetailsVideoEncodingDetailsand have more things available more often \o/. Let's make use of that in to-be-added ui!VideoEncodingDetailschangesVideoEncodingDetailsover time as a bug and will always use the latest foundVideoEncodingDetailsAll this causes some shuffling around. The fact that I changed namespaces a bit didn't help either, but the arbitrary split between
decodeanddemuxdidn't age well. I'll clean this up further in a follow-up.Sideffect, more information from MP4 with H.264:
Before:
After:
