Skip to content

fix(serialization): cap codec allocations - #10340

Merged
ReubenBond merged 2 commits into
dotnet:mainfrom
ReubenBond:fix/cap-codec-allocations
Aug 5, 2026
Merged

ReubenBond merged 2 commits into
dotnet:mainfrom
ReubenBond:fix/cap-codec-allocations

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Aug 4, 2026 •

Copy link
Copy Markdown
Member

Corrupted length prefixes could cause codecs to allocate objects larger than the remaining serialized payload.

Add a shared remaining-byte check to the reader and apply it before allocating strings, buffers, arrays, collections, and protobuf payloads. Keep exception construction off hot paths and cover malformed lengths, forked readers, and non-seekable streams.

Microsoft Reviewers: Open in CodeFlow

Copilot AI review requested due to automatic review settings August 4, 2026 22:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens Orleans serialization deserialization against corrupted length prefixes by introducing a shared “remaining bytes” concept on Reader and using it to validate length-prefixed allocations (strings, buffers, arrays/collections, and protobuf payloads), while also ensuring non-seekable streams don’t break these checks.

Changes:

  • Add Reader.Remaining plus Reader.EnsureAvailable(uint) and apply it to length-prefixed reads/allocation sites.
  • Update many collection/array codecs and protobuf codecs to validate declared lengths against remaining input before allocating.
  • Add/extend unit tests to cover remaining-bytes behavior, forked readers, malformed lengths, and non-seekable streams.
Show a summary per file
File Description
test/Orleans.Serialization.UnitTests/ReaderWriterTests.cs Adds unit tests for Remaining, forked-reader semantics, malformed length rejection, and non-seekable stream behavior.
src/Serializers/Orleans.Serialization.Protobuf/RepeatedFieldCodec.cs Switches repeated-field allocation guard to reader.EnsureAvailable(...).
src/Serializers/Orleans.Serialization.Protobuf/ProtobufCodec.cs Validates protobuf payload length against remaining input before reading/allocating.
src/Serializers/Orleans.Serialization.Protobuf/MapFieldCodec.cs Switches map-field allocation guard to reader.EnsureAvailable(...).
src/Orleans.Serialization/Codecs/StringCodec.cs Ensures string byte-length is available and within supported bounds before reading/allocating.
src/Orleans.Serialization/Codecs/StackCodec.cs Switches stack length guard to reader.EnsureAvailable(...) before allocating backing array.
src/Orleans.Serialization/Codecs/QueueCodec.cs Adds EnsureAvailable guard before allocating queue capacity.
src/Orleans.Serialization/Codecs/MultiDimensionalArrayCodec.cs Adds a remaining-input-based sanity check for declared multi-dimensional array sizes.
src/Orleans.Serialization/Codecs/ListCodec.cs Switches list length guards to reader.EnsureAvailable(...) before allocating/ensuring capacity.
src/Orleans.Serialization/Codecs/HashSetCodec.cs Switches set length guards to reader.EnsureAvailable(...) before allocating/rehydrating.
src/Orleans.Serialization/Codecs/DictionaryCodec.cs Switches dictionary length guards to reader.EnsureAvailable(...) before allocating/rehydrating.
src/Orleans.Serialization/Codecs/CollectionCodec.cs Switches collection length guards to reader.EnsureAvailable(...) before allocating/rehydrating.
src/Orleans.Serialization/Codecs/ByteArrayCodec.cs Adds remaining-input validation for BitArray and PooledBuffer payload lengths.
src/Orleans.Serialization/Codecs/ArrayCodec.cs Switches array/memory/segment length guards to reader.EnsureAvailable(...) before allocating.
src/Orleans.Serialization/Buffers/Reader.cs Introduces Remaining and EnsureAvailable, applies it to byte reads, and updates stream handling for non-seekable inputs.
src/api/Orleans.Serialization/Orleans.Serialization.cs Updates public API surface to include Reader.Remaining and Reader.EnsureAvailable(uint).

Copilot's findings

  • Files reviewed: 16/16 changed files
  • Comments generated: 2

Copilot AI review requested due to automatic review settings August 4, 2026 22:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Suppressed comments (17)

src/Orleans.Serialization/Buffers/Reader.cs:944

  • ReadBytes(uint count) can be passed a declared length > int.MaxValue (e.g., from corrupted length prefixes). In that case it will attempt new byte[count] and later cast count to int, which can overflow/throw or try to allocate an enormous array—especially for non-seekable streams where EnsureAvailable cannot bound the remaining input. Add an explicit maximum-length check and allocate using an int length.
            EnsureAvailable(count);

            var bytes = new byte[count];
            if (IsReadOnlySequenceInput || IsSpanInput || IsBufferSliceInput || IsArcBufferInput)
            {

src/Orleans.Serialization/Codecs/ListCodec.cs:140

  • Same issue as above: (int)UInt32Codec.ReadValue(...) can overflow for encoded lengths > int.MaxValue, and the subsequent (uint)length wrap undermines the guard. Validate encodedLength before casting and before using it for capacity/allocations.
                        var length = (int)UInt32Codec.ReadValue(ref reader, header);
                        reader.EnsureAvailable((uint)length);

src/Orleans.Serialization/Codecs/ArrayCodec.cs:236

  • The declared length is decoded as uint but cast directly to int. If the value is > int.MaxValue, the cast overflows and can lead to invalid allocations or exceptions. Add an explicit max-length check before casting and allocating.
                        length = (int)UInt32Codec.ReadValue(ref reader, header);
                        reader.EnsureAvailable((uint)length);

                        result = new T[length];

src/Orleans.Serialization/Codecs/ArrayCodec.cs:395

  • The length prefix is cast from uint to int without checking > int.MaxValue. Overflow here can produce a negative length and/or huge allocation attempts. Validate the encoded length before casting and allocating.
                        length = (int)UInt32Codec.ReadValue(ref reader, header);
                        reader.EnsureAvailable((uint)length);

                        result = new T[length];
                        ReferenceCodec.RecordObject(reader.Session, result, placeholderReferenceId);

src/Orleans.Serialization/Codecs/ArrayCodec.cs:556

  • The length prefix is cast from uint to int without checking > int.MaxValue. Malformed values can overflow and cause invalid/huge allocation attempts. Validate the encoded value before casting and allocating.
                        length = (int)UInt32Codec.ReadValue(ref reader, header);
                        reader.EnsureAvailable((uint)length);

                        result = new T[length];
                        ReferenceCodec.RecordObject(reader.Session, result, placeholderReferenceId);

src/Orleans.Serialization/Codecs/DictionaryCodec.cs:295

  • Same overflow risk here: the encoded length is cast to int without bounds checking, then used for capacity/constructor arguments. Validate the uint length before casting to avoid negative/overflowed lengths and huge allocation attempts.
                        var length = (int)UInt32Codec.ReadValue(ref reader, header);
                        reader.EnsureAvailable((uint)length);

src/Orleans.Serialization/Codecs/HashSetCodec.cs:160

  • Same overflow risk in the reinitialization path: casting the encoded uint length to int without validating > int.MaxValue can overflow and lead to invalid capacity/constructor arguments. Validate before casting.
                        var length = (int)UInt32Codec.ReadValue(ref reader, header);
                        reader.EnsureAvailable((uint)length);

src/Orleans.Serialization/Codecs/CollectionCodec.cs:143

  • Same overflow risk in the deserialize path: the encoded uint length is cast to int without validating > int.MaxValue. Validate before casting to avoid negative/overflowed lengths affecting control flow.
                case 0:
                    var length = (int)UInt32Codec.ReadValue(ref reader, header);
                    reader.EnsureAvailable((uint)length);

                    break;

src/Orleans.Serialization/Codecs/StackCodec.cs:89

  • The stack length is decoded as uint and cast directly to int. If the encoded value is > int.MaxValue, the cast overflows and can lead to negative lengths and invalid allocations. Validate the encoded length before casting and allocating the backing array.
                case 0:
                    var length = (int)UInt32Codec.ReadValue(ref reader, header);
                    reader.EnsureAvailable((uint)length);

                    array = new T[length];
                    i = length - 1;
                    break;

src/Orleans.Serialization/Codecs/ListCodec.cs:76

  • The length prefix is read as uint and immediately cast to int. Values > int.MaxValue will overflow to a negative length, and the subsequent (uint)length can wrap, potentially bypassing the intended allocation guard or leading to confusing exceptions. Validate the encoded value before casting.

This issue also appears on line 139 of the same file.

                        var length = (int)UInt32Codec.ReadValue(ref reader, header);
                        reader.EnsureAvailable((uint)length);

src/Orleans.Serialization/Codecs/ArrayCodec.cs:85

  • The array length is read as uint then cast to int without bounds checking. Encoded values > int.MaxValue will overflow and can result in attempted huge allocations or inconsistent exceptions. Validate the encoded length before casting/allocating.

This issue also appears in the following locations of the same file:

  • line 233
  • line 391
  • line 552
                        length = (int)UInt32Codec.ReadValue(ref reader, header);
                        reader.EnsureAvailable((uint)length);

                        result = new T[length];

src/Orleans.Serialization/Codecs/DictionaryCodec.cs:109

  • The declared dictionary length is read as uint then cast to int without checking > int.MaxValue. Overflow can produce a negative length and lead to inconsistent exceptions or attempted huge allocations. Validate the encoded value before casting.

This issue also appears on line 293 of the same file.

                        var length = (int)UInt32Codec.ReadValue(ref reader, header);
                        reader.EnsureAvailable((uint)length);

src/Orleans.Serialization/Codecs/HashSetCodec.cs:89

  • The declared set length is cast from uint to int without checking > int.MaxValue. Encoded values above int.MaxValue overflow and can cause negative lengths or large allocation attempts. Validate the encoded length before casting and using it for capacity/allocations.

This issue also appears on line 158 of the same file.

                        var length = (int)UInt32Codec.ReadValue(ref reader, header);
                        reader.EnsureAvailable((uint)length);

                        result = CreateInstance(length, comparer, reader.Session, placeholderReferenceId);

src/Orleans.Serialization/Codecs/CollectionCodec.cs:80

  • The declared collection length is cast from uint to int without checking > int.MaxValue. Overflow can produce a negative length and inconsistent exceptions or attempted huge allocations (e.g., new List<T>(length)). Validate the encoded value before casting.

This issue also appears on line 139 of the same file.

                    var length = (int)UInt32Codec.ReadValue(ref reader, header);
                    reader.EnsureAvailable((uint)length);

                    result = new Collection<T>(new List<T>(length));
                    ReferenceCodec.RecordObject(reader.Session, result, placeholderReferenceId);

src/Orleans.Serialization/Codecs/QueueCodec.cs:82

  • The queue length is cast from uint to int without checking > int.MaxValue. Encoded values above int.MaxValue overflow and can lead to negative lengths or large allocation attempts. Validate the encoded value before casting and using it for capacity.
                    case 0:
                        var length = (int)UInt32Codec.ReadValue(ref reader, header);
                        reader.EnsureAvailable((uint)length);
                        result = new Queue<T>(length);
                        ReferenceCodec.RecordObject(reader.Session, result, placeholderReferenceId);

src/Serializers/Orleans.Serialization.Protobuf/RepeatedFieldCodec.cs:88

  • The RepeatedField length prefix is decoded as uint but cast directly to int. Values > int.MaxValue overflow to a negative length, which can result in invalid capacity settings or inconsistent exceptions. Validate the encoded length before casting.
                    var length = (int)UInt32Codec.ReadValue(ref reader, header);
                    reader.EnsureAvailable((uint)length);

                    result = new RepeatedField<T>{ Capacity = length };
                    ReferenceCodec.RecordObject(reader.Session, result, placeholderReferenceId);

src/Serializers/Orleans.Serialization.Protobuf/MapFieldCodec.cs:97

  • The MapField length prefix is decoded as uint but cast directly to int. Values > int.MaxValue overflow and can lead to invalid allocations/capacity behavior. Validate the encoded value before casting.
                    var length = (int)UInt32Codec.ReadValue(ref reader, header);
                    reader.EnsureAvailable((uint)length);

                    result = CreateInstance(reader.Session, placeholderReferenceId);
                    break;
  • Files reviewed: 16/16 changed files
  • Comments generated: 0 new

Validate declared lengths against the remaining serialized payload before allocating strings, buffers, arrays, and collections.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: dc2895a9-c8ad-41f4-bd7a-cffe1e35e5b0
Replace checked length casts with explicit validation and actionable exceptions before buffer allocation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: dc2895a9-c8ad-41f4-bd7a-cffe1e35e5b0
Copilot AI review requested due to automatic review settings August 5, 2026 01:44
@ReubenBond
ReubenBond force-pushed the fix/cap-codec-allocations branch from 031c907 to 5def074 Compare August 5, 2026 01:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Suppressed comments (3)

src/Orleans.Serialization/Buffers/Reader.cs:483

  • EnsureAvailable relies on Remaining, which returns long.MaxValue for non-seekable streams, so this method will not necessarily throw even when fewer than count bytes are actually available. The doc comment should reflect that it only enforces the check when remaining length can be determined.
        /// <summary>
        /// Throws if fewer than <paramref name="count"/> bytes remain in the input.
        /// </summary>
        /// <param name="count">The number of bytes which are required.</param>

src/Orleans.Serialization/Buffers/Reader.cs:944

  • ReadBytes(uint count) allocates using new byte[count] without guarding against count > int.MaxValue. For corrupted length prefixes on inputs where Remaining is very large (or unknown), this can result in runtime OverflowException/OutOfMemoryException instead of a controlled IndexOutOfRangeException (and may attempt a huge allocation). Consider explicitly rejecting lengths above int.MaxValue and allocating using an int length.
            EnsureAvailable(count);

            var bytes = new byte[count];
            if (IsReadOnlySequenceInput || IsSpanInput || IsBufferSliceInput || IsArcBufferInput)
            {

src/Orleans.Serialization/Buffers/Reader.cs:445

  • The XML doc for Reader<TInput>.Remaining says it returns the number of unread bytes, but for non-seekable streams StreamReaderInput.Remaining returns long.MaxValue. The doc should clarify the "unknown" case to avoid misleading API consumers.

This issue also appears on line 480 of the same file.

        /// <summary>
        /// Gets the number of unread bytes in the input.
        /// </summary>
        public long Remaining
  • Files reviewed: 16/16 changed files
  • Comments generated: 0 new

@ReubenBond
ReubenBond merged commit e473274 into dotnet:main Aug 5, 2026
66 of 67 checks passed
@ReubenBond
ReubenBond deleted the fix/cap-codec-allocations branch August 5, 2026 13:03
This was referenced Aug 28, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants