Arrow IPC encode/decode size guards are asymmetric: large UInt8 payloads encode but are rejected on receive (silent drop)

[phil-opp]

Summary

In the Arrow IPC data plane added by #2366 (81e5efe), the encode guard bounds the input data_len against MAX_IPC_BYTES, but the emitted IPC stream is ~1.125× larger, and every receiver bounds the whole stream against the same MAX_IPC_BYTES. A narrow band of large payloads therefore encodes and is sent successfully, then is rejected by all receivers — silently dropped on the zenoh path.

Details

Encode guards only data_len:

// apis/rust/node/src/node/arrow_utils/ipc_encode.rs:468
fn uint8_layout(data_len: usize) -> eyre::Result<Uint8Layout> {
    if data_len > super::MAX_IPC_BYTES {           // guards the INPUT length
        bail!("UInt8 payload too large: {data_len} bytes ...");
    }
    ...
    let total = schema_block + record_batch_block + body_len + PREFIX_LEN;  // strictly > data_len

total includes the validity bitmap (data_len/8), 64-byte alignment padding, and the schema + record-batch message blocks — roughly 1.125 * data_len. But receivers guard the entire stream:

// apis/rust/node/src/node/arrow_utils.rs:112  decode_arrow_ipc_zero_copy
if buffer.len() > MAX_IPC_BYTES {
    bail!("Arrow IPC payload too large: {} bytes (max {MAX_IPC_BYTES})", buffer.len());
}
// ipc_encode.rs:723  check_ipc_size(len)  — same whole-stream bound

With MAX_IPC_BYTES = 256 MiB, a send_output_raw UInt8 payload with data_len in roughly (227.5 MiB, 256 MiB] passes the encode guard (the producer allocates the buffer and sends it), but the resulting stream exceeds 256 MiB and every receiver bails with "Arrow IPC payload too large." On the zenoh receive path this becomes a warn-and-drop (decode_zenoh_sample → Err), i.e. the input is silently lost; on the daemon path it errors. ipc_fast_path_len can likewise return a value > MAX_IPC_BYTES.

Risk / scope

Niche — single messages that large also tend to collide with SHM-pool / MAX_MESSAGE_BYTES limits in most configurations — but it is a genuine internal inconsistency: the encode limit and the decode limit disagree, and the failure mode on the distributed path is a silent drop rather than a producer-side error.

Suggested fix

Bound the resulting stream length (total / the fast-path length) against MAX_IPC_BYTES in the encode guards, not data_len/the body alone, so that anything the producer accepts is decodable by every receiver (or, conversely, fails loudly at the producer).

---

This issue was filed by a scheduled automated Claude code-review check. The finding was verified against the source at commit caf62a0, but please review before acting.