Repository navigation
feat(journaling): add S3 Express storage provider #10161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
ReubenBond
merged 25 commits into
dotnet:main
from
ReubenBond:rebond/s3-express-journaling
Sep 5, 2026
Merged
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
ace05ab
feat(journaling): add S3 Express storage provider
ReubenBond 0a33925
fix(journaling): address S3 review feedback
ReubenBond 206ff00
test(journaling): add S3 persistence conformance coverage
ReubenBond ce7f646
docs(journaling): document S3 retry defaults
ReubenBond 4633698
test(journaling): cover direct S3 write races
ReubenBond 966c23a
fix(journaling): surface missing S3 buckets
ReubenBond 8761762
fix(journaling): sequence S3 WAL requests
ReubenBond ed26462
fix(journaling): preserve S3 recovery invariants
ReubenBond c3b9030
fix(journaling): validate S3 metadata inputs
ReubenBond 7246d01
fix(journaling): skip unavailable package baseline
ReubenBond 188eb5e
docs(journaling): explain package validation exception
ReubenBond 35cb2da
fix(journaling): cache direct S3 instruments
ReubenBond 15ae4e4
test(journaling): cancel S3 fixture startup
ReubenBond 8af545e
test: cancel S3 persistence adapter gate waits
ReubenBond 52211de
fix(journaling): normalize S3 metadata keys case-insensitively
ReubenBond ae97aaa
test: propagate S3 persistence cancellation
ReubenBond 936281f
fix(journaling): clarify S3 client option contract
ReubenBond 49c3f76
fix(journaling): validate S3 stream read buffer
ReubenBond 134d5fa
fix(journaling): validate S3 rewrite state
ReubenBond 6eb0404
fix(journaling): skip invalid parsed journal ids
ReubenBond 599dc51
fix(journaling): complete synchronous S3 shutdown
ReubenBond f4f45da
ci: isolate S3 journaling provider tests
ReubenBond 86c0567
ci: broaden S3 provider job name
ReubenBond 5c6a274
test(ci): account for S3 provider partition
ReubenBond 4327b98
fix(journaling): clarify null S3 client diagnostic
ReubenBond File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,6 @@ | ||
| { | ||
| "description": "Generated API assemblies which are not currently published as standalone NuGet packages.", | ||
| "packages": {} | ||
| "packages": { | ||
| "Microsoft.Orleans.Journaling.S3": "The new provider is awaiting its first NuGet publication." | ||
| } | ||
| } |
32 changes: 32 additions & 0 deletions
32
src/AWS/Orleans.Journaling.S3/Orleans.Journaling.S3.csproj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <PackageId>Microsoft.Orleans.Journaling.S3</PackageId> | ||
| <PackageReadmeFile>README.md</PackageReadmeFile> | ||
| <TargetFrameworks>$(DefaultTargetFrameworks)</TargetFrameworks> | ||
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <Nullable>enable</Nullable> | ||
| <NoWarn>$(NoWarn);ORLEANSEXP005</NoWarn> | ||
| <VersionSuffix Condition="$(VersionSuffix) != ''">$(VersionSuffix).alpha.1</VersionSuffix> | ||
| <VersionSuffix Condition="$(VersionSuffix) == ''">alpha.1</VersionSuffix> | ||
| <!-- This new package has no published baseline. Enable validation after its first release. --> | ||
| <EnablePackageValidation>false</EnablePackageValidation> | ||
| <OrleansBuildTimeCodeGen>true</OrleansBuildTimeCodeGen> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <PackageReference Include="AWSSDK.S3" /> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <ProjectReference Include="$(SourceRoot)src\Orleans.Journaling\Orleans.Journaling.csproj" /> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <InternalsVisibleTo Include="Orleans.Journaling.Tests" /> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <None Include="README.md" Pack="true" PackagePath="\" /> | ||
| </ItemGroup> | ||
| </Project> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| using System.Diagnostics.CodeAnalysis; | ||
|
|
||
| [assembly: Experimental("ORLEANSEXP005")] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| # Microsoft Orleans S3 Journaling | ||
|
|
||
| `Microsoft.Orleans.Journaling.S3` provides an `IJournalStorage` implementation backed by Amazon S3 Express One Zone directory buckets. | ||
|
|
||
| The provider uses S3 Express append writes (`WriteOffsetBytes`) for WAL appends. For local development and tests against S3-compatible emulators such as MinIO, set `UseS3ExpressAppend = false`, `UseConditionalDelete = false`, and `StorageClass = null` to use portable conditional writes and deletes. | ||
|
|
||
| Buckets should be created ahead of time for AWS S3 Express One Zone. `CreateBucketIfNotExists` is intended for local emulators. | ||
|
|
||
| Metadata updates rewrite the current WAL using a conditional single-object upload. Publish a checkpoint to compact the WAL before updating metadata when the replacement object would exceed S3's 5 GB (5,000,000,000 byte) single-upload limit. Checkpoint snapshots use the same upload limit. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.