Skip to content

feat(journaling): add S3 Express storage provider - #10161

Merged
ReubenBond merged 25 commits into
dotnet:mainfrom
ReubenBond:rebond/s3-express-journaling
Sep 5, 2026
Merged

ReubenBond merged 25 commits into
dotnet:mainfrom
ReubenBond:rebond/s3-express-journaling

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Jun 1, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Add Microsoft.Orleans.Journaling.S3, an Amazon S3 Express One Zone journal storage provider with hosting registration, catalog discovery, metrics, generated API surface, and package documentation.
  • Append WAL records with S3 Express WriteOffsetBytes; provide a conditional rewrite mode for S3-compatible development environments such as MinIO.
  • Exercise the provider through Orleans.Persistence.TestKit using a test-only grain-storage adapter, including generated model-based conformance and same-key optimistic concurrency.

Design

The provider binds every WAL incarnation to a generated identity and uses conditional S3 mutations so stale writers cannot replace or delete a recreated journal. Recovery restores the physical append-part count and requests compaction before the S3 Express part limit is reached.

Checkpoint and metadata publication use conditional single-object writes with explicit 5 GB limits. Definite publication failures reclaim unreferenced checkpoints after manifest reconciliation, while ambiguous outcomes preserve them so a committed WAL never references a deleted checkpoint.

S3 clients created by the provider follow the silo lifecycle; explicitly supplied clients remain caller-owned. Bucket auto-creation is available for local emulators and tolerates concurrent silo initialization. Catalog discovery returns each logical journal identity once across physical key aliases.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

@ReubenBond
ReubenBond force-pushed the rebond/s3-express-journaling branch from 8f517f0 to f85f6dc Compare June 1, 2026 17:38
Copilot AI review requested due to automatic review settings August 18, 2026 09:36
@ReubenBond
ReubenBond force-pushed the rebond/s3-express-journaling branch from d44a00b to cbd9bc0 Compare August 18, 2026 09:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 13/13 changed files
  • Comments generated: 4
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 18, 2026 11:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 14/14 changed files
  • Comments generated: 1
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 18, 2026 12:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/AWS/Orleans.Journaling.S3/S3JournalStorageProvider.cs:113

  • S3JournalStorageProvider initializes an AmazonS3Client but never participates in lifecycle shutdown to dispose it. Since AmazonS3Client holds sockets/HTTP handlers, this can leak resources across silo shutdown/restart. Consider subscribing an onStop handler and disposing the client when Orleans created it (i.e., when options.S3Client was not explicitly provided).
        observer.Subscribe(
            nameof(S3JournalStorageProvider),
            ServiceLifecycleStage.RuntimeInitialize,
            onStart: InitializeAsync);
    }
  • Files reviewed: 14/14 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@ReubenBond

Copy link
Copy Markdown
Member Author

The retried Documentation workflow still fails the package inventory audit because the packable Microsoft.Orleans.Journaling.S3 package is missing from resources/nuget-packages.md (or needs a narrow entry in src/data/package-inventory-exclusions.json). Run: https://github.com/dotnet/orleans/actions/runs/32135336391

Copilot AI review requested due to automatic review settings August 20, 2026 10:04
@ReubenBond
ReubenBond force-pushed the rebond/s3-express-journaling branch from cbfa738 to bedff8c Compare August 20, 2026 10:04
@ReubenBond

Copy link
Copy Markdown
Member Author

Rebased onto current main and fixed the package inventory failure in bedff8c. The public package list now includes Microsoft.Orleans.Journaling.S3, and the unpublished-package/link metadata covers its pre-publication NuGet URL and generated API surface. The source audit, API markdown tests, external-link audit, and full Orleans.slnx build pass; the branch was force-pushed with an exact lease.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 17/17 changed files
  • Comments generated: 1
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 20, 2026 11:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/Orleans.Journaling.Tests/S3JournalStorageTests.cs:4

  • using System.Net; appears to be unused in this test file, which will typically trigger IDE0005 (unused using) and can fail the build when code-style analyzers are enforced as warnings/errors.
using System.Buffers;
using System.Net;
using Amazon;
  • Files reviewed: 17/17 changed files
  • Comments generated: 1
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 20, 2026 11:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/AWS/Orleans.Journaling.S3/S3JournalStorage.cs:562

  • AppendWithS3ExpressAsync constructs a PutObjectRequest without explicitly setting UseChunkEncoding = false, unlike the other PutObjectRequest path in this type (CreatePutObjectRequest). This makes the append path depend on SDK defaults and can change request semantics across SDK versions/endpoints.
                InputStream = stream,
                AutoCloseStream = false,
                IfMatch = expectedETag,
                WriteOffsetBytes = expectedProviderState.ContentLength,
  • Files reviewed: 17/17 changed files
  • Comments generated: 1
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 20, 2026 11:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

src/AWS/Orleans.Journaling.S3/S3JournalStorageProvider.cs:66

  • ListAsync applies the journal-id prefix directly to the S3 object Prefix filter (Prefix = prefix.Value). Since S3JournalStorageOptions allows customizing GetObjectKey/TryParseJournalId, object keys may not start with the journal id value; this server-side prefix filter can therefore omit valid journals and return incomplete results.
        var client = GetClient();
        var bucketName = GetBucketName();
        var objectPrefix = prefix.IsDefault ? null : prefix.Value;
        var journalIds = new List<JournalId>();
        string? continuationToken = null;

        do
        {
            var response = await client.ListObjectsV2Async(
                new ListObjectsV2Request
                {
                    BucketName = bucketName,
                    Prefix = objectPrefix,
                    ContinuationToken = continuationToken,
                },
                cancellationToken).ConfigureAwait(false);

src/AWS/Orleans.Journaling.S3/S3JournalStorage.cs:121

  • UpdateMetadataAsync uses a hard-coded retry count (attempt < 3) for precondition failures, while the rest of the provider uses MaxMetadataOnlyConflictRetries to control in-place retry behavior. This makes metadata updates behave inconsistently and ignores user configuration.
            for (var attempt = 0; attempt < 3; attempt++)
            {
                var properties = await GetPropertiesCoreAsync(expectedETag: null, cancellationToken).ConfigureAwait(false);

src/AWS/Orleans.Journaling.S3/S3JournalStorage.cs:563

  • AppendWithS3ExpressAsync constructs a PutObjectRequest without setting UseChunkEncoding=false, while other PutObjectRequest instances created by this provider explicitly disable chunked encoding. This inconsistency can lead to different request signing/transport behavior between append vs create/replace paths.
        var response = await _client.PutObjectAsync(
            new PutObjectRequest
            {
                BucketName = _shared.BucketName,
                Key = _walObjectKey,
                InputStream = stream,
                AutoCloseStream = false,
                IfMatch = expectedETag,
                WriteOffsetBytes = expectedProviderState.ContentLength,
            },
  • Files reviewed: 17/17 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

It introduces a substantial new storage provider with non-trivial correctness/concurrency semantics plus new CI partitions, so it warrants final human validation before approval.

Review tier: Lite
Findings: None

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

It introduces a substantial new storage provider with intricate correctness/concurrency and backend-specific behavior which warrants final human review despite strong test coverage.

Review tier: Lite
Findings: None

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants