Repository navigation
feat(journaling): add S3 Express storage provider - #10161
Conversation
8f517f0 to
f85f6dc
Compare
d44a00b to
cbd9bc0
Compare
There was a problem hiding this comment.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
src/AWS/Orleans.Journaling.S3/S3JournalStorageProvider.cs:113
- S3JournalStorageProvider initializes an AmazonS3Client but never participates in lifecycle shutdown to dispose it. Since AmazonS3Client holds sockets/HTTP handlers, this can leak resources across silo shutdown/restart. Consider subscribing an onStop handler and disposing the client when Orleans created it (i.e., when options.S3Client was not explicitly provided).
observer.Subscribe(
nameof(S3JournalStorageProvider),
ServiceLifecycleStage.RuntimeInitialize,
onStart: InitializeAsync);
}
- Files reviewed: 14/14 changed files
- Comments generated: 0 new
- Review effort level: Lite
|
The retried Documentation workflow still fails the package inventory audit because the packable |
cbfa738 to
bedff8c
Compare
|
Rebased onto current main and fixed the package inventory failure in bedff8c. The public package list now includes Microsoft.Orleans.Journaling.S3, and the unpublished-package/link metadata covers its pre-publication NuGet URL and generated API surface. The source audit, API markdown tests, external-link audit, and full Orleans.slnx build pass; the branch was force-pushed with an exact lease. |
There was a problem hiding this comment.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
test/Orleans.Journaling.Tests/S3JournalStorageTests.cs:4
using System.Net;appears to be unused in this test file, which will typically trigger IDE0005 (unused using) and can fail the build when code-style analyzers are enforced as warnings/errors.
using System.Buffers;
using System.Net;
using Amazon;
- Files reviewed: 17/17 changed files
- Comments generated: 1
- Review effort level: Lite
There was a problem hiding this comment.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
src/AWS/Orleans.Journaling.S3/S3JournalStorage.cs:562
AppendWithS3ExpressAsyncconstructs aPutObjectRequestwithout explicitly settingUseChunkEncoding = false, unlike the otherPutObjectRequestpath in this type (CreatePutObjectRequest). This makes the append path depend on SDK defaults and can change request semantics across SDK versions/endpoints.
InputStream = stream,
AutoCloseStream = false,
IfMatch = expectedETag,
WriteOffsetBytes = expectedProviderState.ContentLength,
- Files reviewed: 17/17 changed files
- Comments generated: 1
- Review effort level: Lite
There was a problem hiding this comment.
Review details
Suppressed comments (3)
Previously missed (3) — in code that hasn't changed since the last review.
src/AWS/Orleans.Journaling.S3/S3JournalStorageProvider.cs:66
- ListAsync applies the journal-id prefix directly to the S3 object Prefix filter (Prefix = prefix.Value). Since S3JournalStorageOptions allows customizing GetObjectKey/TryParseJournalId, object keys may not start with the journal id value; this server-side prefix filter can therefore omit valid journals and return incomplete results.
var client = GetClient();
var bucketName = GetBucketName();
var objectPrefix = prefix.IsDefault ? null : prefix.Value;
var journalIds = new List<JournalId>();
string? continuationToken = null;
do
{
var response = await client.ListObjectsV2Async(
new ListObjectsV2Request
{
BucketName = bucketName,
Prefix = objectPrefix,
ContinuationToken = continuationToken,
},
cancellationToken).ConfigureAwait(false);
src/AWS/Orleans.Journaling.S3/S3JournalStorage.cs:121
- UpdateMetadataAsync uses a hard-coded retry count (attempt < 3) for precondition failures, while the rest of the provider uses MaxMetadataOnlyConflictRetries to control in-place retry behavior. This makes metadata updates behave inconsistently and ignores user configuration.
for (var attempt = 0; attempt < 3; attempt++)
{
var properties = await GetPropertiesCoreAsync(expectedETag: null, cancellationToken).ConfigureAwait(false);
src/AWS/Orleans.Journaling.S3/S3JournalStorage.cs:563
- AppendWithS3ExpressAsync constructs a PutObjectRequest without setting UseChunkEncoding=false, while other PutObjectRequest instances created by this provider explicitly disable chunked encoding. This inconsistency can lead to different request signing/transport behavior between append vs create/replace paths.
var response = await _client.PutObjectAsync(
new PutObjectRequest
{
BucketName = _shared.BucketName,
Key = _walObjectKey,
InputStream = stream,
AutoCloseStream = false,
IfMatch = expectedETag,
WriteOffsetBytes = expectedProviderState.ContentLength,
},
- Files reviewed: 17/17 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
It introduces a substantial new storage provider with non-trivial correctness/concurrency semantics plus new CI partitions, so it warrants final human validation before approval.
Review tier: Lite
Findings: None
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
It introduces a substantial new storage provider with intricate correctness/concurrency and backend-specific behavior which warrants final human review despite strong test coverage.
Review tier: Lite
Findings: None
Summary
Microsoft.Orleans.Journaling.S3, an Amazon S3 Express One Zone journal storage provider with hosting registration, catalog discovery, metrics, generated API surface, and package documentation.WriteOffsetBytes; provide a conditional rewrite mode for S3-compatible development environments such as MinIO.Orleans.Persistence.TestKitusing a test-only grain-storage adapter, including generated model-based conformance and same-key optimistic concurrency.Design
The provider binds every WAL incarnation to a generated identity and uses conditional S3 mutations so stale writers cannot replace or delete a recreated journal. Recovery restores the physical append-part count and requests compaction before the S3 Express part limit is reached.
Checkpoint and metadata publication use conditional single-object writes with explicit 5 GB limits. Definite publication failures reclaim unreferenced checkpoints after manifest reconciliation, while ambiguous outcomes preserve them so a committed WAL never references a deleted checkpoint.
S3 clients created by the provider follow the silo lifecycle; explicitly supplied clients remain caller-owned. Bucket auto-creation is available for local emulators and tolerates concurrent silo initialization. Catalog discovery returns each logical journal identity once across physical key aliases.