Video: unify rendering on videoconvert→appsink + cross-platform zero-copy#14332
Merged
Conversation
Contributor
Build ResultsPlatform Status
All builds passed. Pre-commit
Pre-commit hooks: 4 passed, 44 failed, 7 skipped. Test Resultslinux-coverage: 90 passed, 0 skipped Code Coverage
Artifact Sizes
Updated: 2026-05-09 06:29:19 UTC • Triggered by: Android |
HTRamsey
force-pushed
the
refactor/video-appsink
branch
4 times, most recently
from
81018d4 to
bfc8da4
Compare
HTRamsey
force-pushed
the
refactor/video-appsink
branch
2 times, most recently
from
1a05440 to
16cea1b
Compare
HTRamsey
force-pushed
the
refactor/video-appsink
branch
2 times, most recently
from
a4982c4 to
96cec4b
Compare
Collaborator
Author
|
It's basically an adaptation of this for each platform: https://github.com/qt/qtmultimedia/blob/dev/src/plugins/multimedia/gstreamer/common/qgstvideobuffer.cpp |
HTRamsey
force-pushed
the
refactor/video-appsink
branch
3 times, most recently
from
914b174 to
29f876c
Compare
Contributor
|
I tested on both Ubuntu 24.04 and Android and it seems to be working well on my end. |
HTRamsey
force-pushed
the
refactor/video-appsink
branch
6 times, most recently
from
a31a92f to
754cefc
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Unifies QGroundControl’s GStreamer video rendering by removing the custom per-platform Qt6 QML sink plugins and routing all decoded frames through a single qgcvideosinkbin (videoconvert/glupload → appsink) into Qt’s VideoOutput via a new GstAppSinkAdapter, with optional per-platform zero-copy GPU import paths and CPU fallback.
Changes:
- Replaces
qml6glsink/qml6d3d11sinkusage with a single in-treeqgcvideosinkbin+appsink→QVideoSinkadapter flow. - Adds cross-platform GPU zero-copy infrastructure (GLMemory/DMABuf/D3D11/D3D12/IOSurface/AHardwareBuffer) plus context-bridge plumbing and telemetry.
- Updates QML/UI/settings/tests to use
QtMultimedia.VideoOutputand introduces theforceCpuVideoPathsetting (with legacy migration).
Reviewed changes
Copilot reviewed 87 out of 87 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/setup/build-gstreamer.py | Updates CLI help text for Qt prefix usage. |
| test/VideoManager/GStreamer/GStreamerTest.h | Adds declarations for expanded GStreamer/appsink/zero-copy tests. |
| test/VideoManager/GStreamer/CMakeLists.txt | Adjusts test target notes/documentation. |
| test/Camera/VideoManagerTest.h | Renames unit test to match new QML type expectation. |
| test/Camera/VideoManagerTest.cc | Switches unit test QML from custom Gst item to QtMultimedia.VideoOutput. |
| src/VideoManager/VideoReceiver/GStreamer/VideoItemStub.h | Removes stub QML item previously used for custom sink types. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/QGCRhiCapture.h | Adds helper API to capture/cache the live QRhi*. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/QGCRhiCapture.cc | Implements QRhi caching and bridge reset on scene graph teardown. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstIOSurfaceVideoBuffer.mm | Adds IOSurface/CVPixelBuffer zero-copy buffer wrapper (Metal). |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstIOSurfaceVideoBuffer.h | Declares IOSurface zero-copy buffer wrapper. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstHwVideoBufferFactory.h | Declares factory for selecting per-sample GPU buffer wrappers. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstHwVideoBufferFactory.cc | Implements dispatch to DMABuf/GL/D3D/IOSurface/AHB wrappers with CPU fallback. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstHwVideoBuffer.h | Introduces common base class for GStreamer-backed QHwVideoBuffers. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstHwVideoBuffer.cc | Implements GstSample lifetime ownership for the base buffer class. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstGlVideoBuffer.h | Declares GLMemory zero-copy buffer wrapper. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstGlVideoBuffer.cc | Implements GLMemory texture wrapping + sync handling. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstGlContextBridge.h | Declares Qt↔gst-gl context bridging API. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstGlContextBridge.cc | Implements NEED_CONTEXT/query handling and bridge registration/reset. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstDmaBufVideoBuffer.h | Declares DMABuf→EGLImage zero-copy wrapper. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstDmaBufVideoBuffer.cc | Implements DMABuf import (modifier-aware) and texture lifecycle. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstD3DVideoBufferCommon.h | Adds shared D3D11/D3D12 frame-texture scaffolding and diagnostics. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstD3DContextBridgeCommon.h | Declares shared D3D context bridge helpers. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstD3DContextBridgeCommon.cc | Implements shared QRhi backend checks and NEED_CONTEXT matching/logging. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstD3D12VideoBuffer.h | Declares D3D12 zero-copy buffer wrapper. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstD3D12VideoBuffer.cc | Implements D3D12 resource import with slice staging when required. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstD3D12ContextBridge.h | Declares D3D12 device bridging for gst-d3d12 elements. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstD3D12ContextBridge.cc | Implements D3D12 device creation/dispatch and bridge registration/reset. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstD3D11VideoBuffer.h | Declares D3D11 zero-copy buffer wrapper. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstD3D11VideoBuffer.cc | Implements D3D11 texture import with slice staging when required. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstD3D11ContextBridge.h | Declares D3D11 device bridging for gst-d3d11 elements. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstD3D11ContextBridge.cc | Implements D3D11 device wrapping/dispatch and bridge registration/reset. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstContextBridgeRegistry.h | Declares registry to fan out GstBus sync messages to bridges. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstContextBridgeRegistry.cc | Implements registry, reset fanout, and test-only clearing. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstAHardwareBufferVideoBuffer.h | Declares Android AHardwareBuffer zero-copy wrapper. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/GstAHardwareBufferVideoBuffer.cc | Implements AHardwareBuffer import via EGLImage/ExternalOES. |
| src/VideoManager/VideoReceiver/GStreamer/HwBuffers/CMakeLists.txt | Adds build-time feature detection and per-platform GPU-path compilation. |
| src/VideoManager/VideoReceiver/GStreamer/GstVideoReceiver.h | Adds decoder/QoS stats properties and probe state to receiver. |
| src/VideoManager/VideoReceiver/GStreamer/GStreamerLogging.cc | Renames logging categories for consistency. |
| src/VideoManager/VideoReceiver/GStreamer/GStreamerHelpers.cc | Renames logging category and updates decoder-rank comment. |
| src/VideoManager/VideoReceiver/GStreamer/GStreamer.h | Renames and generalizes sink adapter setup API. |
| src/VideoManager/VideoReceiver/GStreamer/GStreamer.cc | Switches to appsink-based sinkbin creation and shared adapter setup; updates plugin checks. |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6gl/qt6/qt6glitem.h | Removes custom Qt6 GL QML sink code (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6gl/qt6/gstqt6glutility.h | Removes custom Qt6 GL QML sink utilities (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6gl/qt6/gstqt6glutility.cc | Removes custom Qt6 GL QML sink utilities impl (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6gl/qt6/gstqt6gl.h | Removes custom Qt6 GL QML sink header (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6gl/qt6/gstqt6elements.h | Removes custom Qt6 GL QML sink elements header (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6gl/qt6/gstqt6element.cc | Removes custom Qt6 GL QML sink registration (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6gl/qt6/gstqsg6glnode.h | Removes GL scenegraph node for old sink (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6gl/qt6/gstqsg6glnode.cc | Removes GL scenegraph node impl for old sink (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6gl/qt6/gstqml6glsink.h | Removes old qml6gl sink header (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6gl/qt6/gstplugin.cc | Removes old qml6gl plugin glue (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6gl/gstqml6glregister.h | Removes old QML type registration shim (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6gl/gstqml6glregister.cpp | Removes old QML type registration shim (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6gl/CMakeLists.txt | Removes build for old qml6gl plugin (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6d3d11/gstqml6d3d11register.h | Removes old D3D11 QML type registration shim (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6d3d11/gstqml6d3d11register.cpp | Removes old D3D11 QML type registration shim (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqml6d3d11/CMakeLists.txt | Removes build for old qt6d3d11 sink plugin (deleted). |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqgcvideosinkbin.h | Updates sink bin struct and adds appsink accessor API. |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqgcelements.h | Adds include guard + exports debug category pointer. |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqgcelement.cc | Switches debug category from static to exported definition. |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/gstqgc.cc | Updates plugin init and versioning to use QGC app version. |
| src/VideoManager/VideoReceiver/GStreamer/gstqgc/CMakeLists.txt | Stops building old sink plugins; documents static registration model. |
| src/VideoManager/VideoReceiver/GStreamer/GstAppSinkAdapter.h | Expands adapter API: telemetry properties, caps cache helpers, GPU-path plumbing. |
| src/VideoManager/VideoReceiver/GStreamer/CMakeLists.txt | Adjusts GStreamer deps and wires appsink adapter + HwBuffers subdir. |
| src/VideoManager/VideoManager.h | Removes per-platform sink capability properties from API. |
| src/VideoManager/VideoManager.cc | Removes old QML sink registration/stubs; wires QRhi capture and unified adapter setup. |
| src/UI/AppSettings/pages/Video.SettingsUI.json | Adds UI control for forceCpuVideoPath. |
| src/Settings/VideoSettings.h | Adds forceCpuVideoPath setting fact. |
| src/Settings/VideoSettings.cc | Migrates legacy zero-copy setting and exposes new force-CPU setting visibility. |
| src/Settings/Video.SettingsGroup.json | Adds forceCpuVideoPath setting metadata and descriptions. |
| src/QGCApplication.cc | Removes OpenGL probing logic; keeps default graphics API for appsink→VideoOutput path. |
| src/FlyView/QGCVideoBackgroundD3D11.qml | Removes old D3D11-specific video background (deleted). |
| src/FlyView/QGCVideoBackground.qml | Removes old GL-specific video background (deleted). |
| src/FlyView/FlightDisplayViewVideoOutput.qml | Adds VideoOutput-based FlyView video component and fillMode mapping. |
| src/FlyView/FlightDisplayViewVideo.qml | Switches loaders to the unified VideoOutput component for all platforms. |
| src/FlyView/FlightDisplayViewGStreamerD3D11.qml | Removes old custom D3D11 Gst QML item usage (deleted). |
| src/FlyView/FlightDisplayViewGStreamer.qml | Removes old custom GL Gst QML item usage (deleted). |
| src/FlyView/CMakeLists.txt | Updates QML module file list to remove old components and add VideoOutput component. |
| cmake/platform/Apple.cmake | Enables OBJC/OBJCXX at root scope for .mm compilation. |
| cmake/find-modules/FindQGCGStreamer.cmake | Updates iOS discovery, plugin lists, xcframework handling, and feature probes. |
| cmake/find-modules/FindGStreamer.cmake | Fixes component re-find behavior and makes imported targets GLOBAL. |
Comments suppressed due to low confidence (1)
cmake/find-modules/FindQGCGStreamer.cmake:629
- The xcframework branch sets
GStreamer_USE_XCFRAMEWORKand the slice paths, but then falls through into the “Classic .framework path” block which re-normalizes paths and setsGStreamer_USE_FRAMEWORK, likely clobbering the xcframework configuration. This used toreturn(); consider restoring the early-exit (or wrapping the classic path in anelse()) so xcframework and framework discovery don’t both run.
_gst_normalize_and_validate_root()
endif()
# ── Classic .framework path ───────────────────────────────────────────────
_gst_normalize_and_validate_root()
Comment on lines
+862
to
+867
| if (GstElementFactory *factory = gst_element_factory_find("qgcvideosinkbin")) { | ||
| videoSinkBin = gst_element_factory_create_full(factory, | ||
| "gpu-zerocopy", gpuZeroCopy ? TRUE : FALSE, | ||
| NULL); | ||
| gst_object_unref(factory); | ||
| } |
Comment on lines
+216
to
+220
| const EGLint attribs[] = { EGL_NONE }; | ||
| EGLImageKHR image = eglCreateImageKHR_(_eglDisplay, EGL_NO_CONTEXT, | ||
| EGL_NATIVE_BUFFER_ANDROID, | ||
| clientBuffer, attribs); | ||
| if (image == EGL_NO_IMAGE_KHR) { |
HTRamsey
force-pushed
the
refactor/video-appsink
branch
8 times, most recently
from
0459bfa to
07c50ad
Compare
Drop the vendored qml6glsink and qml6d3d11sink modules (and the gst-plugins-good / gst-plugins-bad source downloads they required) and funnel all GStreamer rendering through a single videoconvert→appsink → GstAppSinkAdapter → QVideoSink → QML VideoOutput chain. Qt picks the native RHI backend (Metal / Vulkan / D3D11 / OpenGL), so QGCApplication no longer probes for an OpenGL context or forces a graphics API. qgcvideosinkbin loses the d3d11sink and glsinkbin fallbacks plus the widget / aspect-ratio properties they exposed. GStreamer.cc drops the qml6 / qt6d3d11 / opengl static plugin registrations and verifies only appsink / qgc / coreelements; the iOS xcframework no longer needs the opengl plugin filter. VideoManager removes gstreamerD3D11Sink and gstreamerAppleSink (the QML side now picks one VideoOutput component unconditionally), and FlightDisplayViewMetal.qml is renamed to FlightDisplayViewVideoOutput.qml to reflect the universal path.
HTRamsey
force-pushed
the
refactor/video-appsink
branch
from
07c50ad to
7dafa79
Compare
Contributor
|
Can we find a way to not rely on the |
23 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the per-platform custom GStreamer sinks (
qml6glsink,qml6d3d11sink) with a singleqgcvideosinkbinthat pushes frames to aQVideoSinkviaappsink+ a thin adapter (GstAppSinkAdapter). Adds optional zero-copy GPU paths (DMABuf/EGLImage on Linux, D3D11 on Windows, IOSurface on macOS, AHardwareBuffer on Android) with automatic per-frame fallback to a CPU copy if a GPU import fails.The Linux GPU path routes through
gluploadso va'sDMA_DRMDMABuf output is imported into GL textures (still zero-copy) and consumed asGLMemoryby appsink — works on GStreamer 1.24+ with the Mesa va plugin without needing 1.26'sformat=DMA_DRMappsink negotiation.Why
gstqml6gl/, nogstqml6d3d11/).GstAppSinkAdapter, which makes telemetry, fallback, and lifecycle uniform across receivers.What changes
Pipeline
qgcvideosinkbin(LGPL, ships in-tree): one bin with two construction modes selected by a construct-onlygpu-zerocopyGObject prop.gpu-zerocopy=FALSE):videoconvert → appsinkaccepting widely-supported formats. Always works.gpu-zerocopy=TRUE): platform-specific zero-copy. On Linux:glupload → appsinkwithmemory:GLMemorycaps. On other platforms: bareappsinkwith the platform's native memory feature in caps (memory:D3D11Memory,memory:AHardwareBuffer, etc.).qml6glsink,qml6d3d11sink) removed.Adapter (
GstAppSinkAdapter)QHwVideoBuffersubclass viamakeHwVideoBuffer()factory.nullptr⇒ adapter copies to system memory (CPU fallback path) instead of crashing.Q_PROPERTYtelemetry:gpuFrameCount,cpuFrameCount,gpuFallbackCountfor runtime introspection (1 Hz throttled signal).gst_video_info_dma_drm_from_capsso future decoders that emit them work without further changes.gpu-zerocopyback from the bin viag_object_getafter construction so the adapter's path mode can never disagree with the bin's actual pipeline (eliminates a class of fact-vs-bin desync bugs).QHwVideoBufferimplementationsGstDmaBufVideoBuffer— Linux DMABuf via EGLImage import, modifier-aware (EGL_EXT_image_dma_buf_import_modifiers).GstGlVideoBuffer— generic GLMemory wrap with properGstGLSyncMetaflush before texture access.GstD3D11VideoBuffer— Windows D3D11Memory share with QRhi's D3D11 backend; ref-leak-safe on early-exit.GstIOSurfaceVideoBuffer— macOS CoreVideo CVPixelBuffer viagst_apple_core_video_memory_get_pixel_buffer()(replaces a UB pointer-cast heuristic).GstAHardwareBufferVideoBuffer— Android AHardwareBuffer viaeglGetNativeClientBufferANDROID(gst-plugins-bad ≥ 1.28).GstHwVideoBufferbase +GstHwVideoBufferFactoryconsolidate the previously copy-pasted GstSample/info/format ownership and dispatch.Context bridges
GstGlContextBridge— Linux GL context bridge for glupload'sNEED_CONTEXTbus message. Tries Qt's EGL native interface first (Wayland/eglfs/xcb_egl), falls back to GLX (the xcb default on Qt 6 desktop) wrappingGstGLDisplayX11+GST_GL_PLATFORM_GLX.GstD3D11ContextBridge— Windows D3D11 device wrap viagst_d3d11_device_wrap.GstContextBridgeRegistry— replaces a hardcoded#ifchain inGstVideoReceiverwith a registry-based dispatcher.Settings
gpuZeroCopyEnabled(default off, "experimental") replaced withforceCpuVideoPath(default off, no "experimental" framing). Default behavior is now: use GPU when the decoder supports it, fall back to CPU per-frame on import failure. The new flag is a debug escape hatch that forces the CPU path.forceCpuVideoPath=trueandforceVideoDecoder=Softwareboth resolve to the CPU pipeline. Software decoder + GPU path is structurally impossible (SW decoders output system memory) — handled silently inGStreamer::createVideoSink, no warning needed.Tests
QGCRhiCapturecache lifecycle, CPU memcpy active-row stride correctness.Verified on Linux desktop (xcb, GStreamer 1.24.2)
RTSP H.265 4K @ 10 fps from a Dahua camera, vah265dec selected by decodebin3:
glupload → appsinkGLMemoryPBOGL:627/0forceCpuVideoPath=truevideoconvert → appsinkSystemMemoryCPU:387forceVideoDecoder=Softwarevideoconvert → appsinkSystemMemoryCPU:nStart/stop exercised across multiple process cycles; clean teardown (
Adapter teardown — CPU:0 DMABuf:0 GL:627 GL-failures:0), no leaks reported.Performance
Same throughput across all three modes when the camera is the bottleneck. Differences scale with resolution and stream count:
The GPU path is what makes multi-stream viable; per-stream marginal cost on the GPU path is essentially decode + render budget, with no cross-bus traffic.
Test plan
forceCpuVideoPath=true+forceVideoDecoder=Software