FFMPEG Core - Video Decoding Modification

[kaysedwards]

I changed the way video decoding works slightly in the FFMPEG core to, hopefully, enable CODECs that don't play well--such as AV1 in my experience--with the current implementation.

The changes I've made have been done to more precisely follow the documentation of the FFMPEG developers with respect to swapping between sending packets and receiving frames.

The bulk of the change allows packet sending and frame receiving to accumulate enough data to decode a frame without going overboard on the memory requirements.

Note: A serious bug exists in the current implementation that breaks playback with files having rates that vary in different streams; my patch does nothing to address that bug; my patch only changes the decoding behavior and fixes a user-facing message for accuracy.

[hizzlekizzle]

Hmm, it's crashing about a half-second into the video for me, which is the same as upstream master, using this test video: https://test-videos.co.uk/vids/sintel/mp4/av1/1080/Sintel_1080_10s_30MB.mp4

[kaysedwards]

I did use that video for testing as we discussed.

I wonder:

Did you do a full rebuild?
Did you build as a core?
Did you build as internal?

I ask because during development I tried doing a retroarch -L ffmpeg test.mp4 and it would reliably use the internal version.

[kaysedwards]

Hmm. Yeah. It is working on my end. (https://streamable.com/hwn7fm)

I just cloned my repository; I did the configure, make, and make install dance; I then played the reference video just fine.

[hizzlekizzle]

oh yeah, weird. if I --disable-ffmpeg, it'll load the core just fine, but if it has internal ffmpeg, it tries to use that instead. Anyway, your version indeed plays the whole thing. LGTM,then!

[kaysedwards]

oh yeah, weird. if I --disable-ffmpeg, it'll load the core just fine, but if it has internal ffmpeg, it tries to use that instead. Anyway, your version indeed plays the whole thing. LGTM,then!

Yeah. I... confess that it took me way longer than it should have to figure out what was happening.

[LibretroAdmin]

This unfortunately causes a lot of my video files to no longer load

[libretro ERROR] [FFMPEG] Error transferring the data to system memory: Invalid argument
Assertion desc failed at D:/W/B/src/ffmpeg-8.0/libswscale/swscale_internal.h:758

For now I'll be reverting this. However, you can submit a new PR that is hopefully more robust and tested.
My advice would be to make small changes in several PRs instead of one big PR, otherwise it's almost impossible to pinpoint what is causing the issues

[kaysedwards]

However, you can submit a new PR that is hopefully more robust and tested. My advice would be to make small changes in several PRs instead of one big PR, otherwise it's almost impossible to pinpoint what is causing the issues

Do you always talk down to people who just want to contribute?

Did you even look at exactly how much code had changed before posting with such condescension?

I don't appreciate the attitude.

That said, I will thank you for the bug report; the issue is born or my misreading the commit I worked from for the patch; I'm not sure how or when I got it wrong during the modifications, but the fix was reasonably found by looking at the code side-by-side: the av_hwframe_transfer_data function must be called during a different part of the process than what is done in my patch during decoding.

I don't have AV1 decoding hardware available just now so more work may still need to be done. I'd appreciate any test you can run assuming you have access to the hardware.

For whatever (I don't really use Github as a base.) reason, my code can't be merged automatically so some manual work must apparently be done in either event. I can make a pull request as is or I can fix things on my side.

[hizzlekizzle]

I don't think he meant anything by it. I should have tested it more, myself.

[kaysedwards]

Well, I can't get any eyes on this patch from people who have the hardware.

I guess the enhanced CODEC support is dead in the water.

[hizzlekizzle]

is the patch in your fork? I can try to do some testing this evening if it's ready for stress.

[kaysedwards]

Yes. The latest version is in my Github repository. Thanks!

I'm not sure you have the hardware available considering discussion elsewhere, but I'd appreciate anyone testing who does have the hardware.

If you get apparently spurious warnings/errors, please check with ffmpeg proper: ffmpeg -v error -f null - -i file.ext

We are doing nothing to prevent those warnings/errors in the logs, and I'm not certain anything even could be done due to the nature of the library.

A good example might be the AV1 encoded Sintel clip earlier discussed; the 30MiB 1080p variant, at least, produces several "Invalid data found when processing input" errors.