Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
ace05ab
feat(journaling): add S3 Express storage provider
ReubenBond Aug 21, 2026
0a33925
fix(journaling): address S3 review feedback
ReubenBond Aug 23, 2026
206ff00
test(journaling): add S3 persistence conformance coverage
ReubenBond Aug 23, 2026
ce7f646
docs(journaling): document S3 retry defaults
ReubenBond Aug 23, 2026
4633698
test(journaling): cover direct S3 write races
ReubenBond Aug 23, 2026
966c23a
fix(journaling): surface missing S3 buckets
ReubenBond Aug 23, 2026
8761762
fix(journaling): sequence S3 WAL requests
ReubenBond Aug 23, 2026
ed26462
fix(journaling): preserve S3 recovery invariants
ReubenBond Aug 25, 2026
c3b9030
fix(journaling): validate S3 metadata inputs
ReubenBond Sep 4, 2026
7246d01
fix(journaling): skip unavailable package baseline
ReubenBond Sep 4, 2026
188eb5e
docs(journaling): explain package validation exception
ReubenBond Sep 4, 2026
35cb2da
fix(journaling): cache direct S3 instruments
ReubenBond Sep 4, 2026
15ae4e4
test(journaling): cancel S3 fixture startup
ReubenBond Sep 4, 2026
8af545e
test: cancel S3 persistence adapter gate waits
ReubenBond Sep 4, 2026
52211de
fix(journaling): normalize S3 metadata keys case-insensitively
ReubenBond Sep 5, 2026
ae97aaa
test: propagate S3 persistence cancellation
ReubenBond Sep 5, 2026
936281f
fix(journaling): clarify S3 client option contract
ReubenBond Sep 5, 2026
49c3f76
fix(journaling): validate S3 stream read buffer
ReubenBond Sep 5, 2026
134d5fa
fix(journaling): validate S3 rewrite state
ReubenBond Sep 5, 2026
6eb0404
fix(journaling): skip invalid parsed journal ids
ReubenBond Sep 5, 2026
599dc51
fix(journaling): complete synchronous S3 shutdown
ReubenBond Sep 5, 2026
f4f45da
ci: isolate S3 journaling provider tests
ReubenBond Sep 5, 2026
86c0567
ci: broaden S3 provider job name
ReubenBond Sep 5, 2026
5c6a274
test(ci): account for S3 provider partition
ReubenBond Sep 5, 2026
4327b98
fix(journaling): clarify null S3 client diagnostic
ReubenBond Sep 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/scripts/test-summarize-coverage.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -1235,8 +1235,8 @@ exit 0
$workflow = Get-Content -Raw -LiteralPath $workflowPath
$runTestsAction = Get-Content -Raw -LiteralPath $runTestsActionPath
$dotnetTestAction = Get-Content -Raw -LiteralPath $dotnetTestActionPath
Assert-Equal 18 ([regex]::Matches($workflow, 'uses: \./\.github/actions/run-tests')).Count 'Test action count differs.'
Assert-Equal 16 ([regex]::Matches($workflow, '(?m)^\s{8}provider: [A-Za-z]')).Count 'Provider-discovered test partition count differs.'
Assert-Equal 19 ([regex]::Matches($workflow, 'uses: \./\.github/actions/run-tests')).Count 'Test action count differs.'
Assert-Equal 17 ([regex]::Matches($workflow, '(?m)^\s{8}provider: [A-Za-z]')).Count 'Provider-discovered test partition count differs.'
Assert-Equal 2 ([regex]::Matches($runTestsAction, 'uses: \./\.github/actions/dotnet-test')).Count 'Native test action invocation count differs.'
Assert-Equal 2 ([regex]::Matches($runTestsAction, "format\('/\[\(Provider=\{0\}\)")).Count 'Standard provider filter count differs.'
$directTestCommands = ([regex]::Matches($dotnetTestAction, 'dotnet test --solution Orleans\.slnx')).Count
Expand Down
15 changes: 15 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -578,6 +578,21 @@ jobs:
- name: Clean up DynamoDB Local
if: always()
run: docker rm -f dynamodb || true
test-s3:
name: AWS S3 provider tests
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
framework: ["net8.0", "net10.0"]
steps:
- uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5
- name: Test
uses: ./.github/actions/run-tests
with:
artifact-name: test_output_${{ github.job }}_${{ matrix.framework }}
framework: ${{ matrix.framework }}
provider: S3
test-sqs:
name: AWS SQS provider tests
runs-on: ubuntu-latest
Expand Down
1 change: 1 addition & 0 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
<PackageVersion Include="AwesomeAssertions" Version="9.3.0" />
<PackageVersion Include="AWSSDK.DynamoDBv2" Version="4.0.14" />
<PackageVersion Include="AWSSDK.Kinesis" Version="4.0.2.12" />
<PackageVersion Include="AWSSDK.S3" Version="4.0.23.4" />
<PackageVersion Include="AWSSDK.SQS" Version="4.0.2.14" />
<PackageVersion Include="Azure.Core" Version="1.50.0" />
<PackageVersion Include="Azure.Data.Tables" Version="12.11.0" />
Expand Down
1 change: 1 addition & 0 deletions Orleans.slnx
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@
</Folder>
<Folder Name="/src/Extensions/AWS/">
<Project Path="src/AWS/Orleans.Clustering.DynamoDB/Orleans.Clustering.DynamoDB.csproj" />
<Project Path="src/AWS/Orleans.Journaling.S3/Orleans.Journaling.S3.csproj" />
<Project Path="src/AWS/Orleans.Persistence.DynamoDB/Orleans.Persistence.DynamoDB.csproj" />
<Project Path="src/AWS/Orleans.Reminders.DynamoDB/Orleans.Reminders.DynamoDB.csproj" />
<Project Path="src/AWS/Orleans.Streaming.Kinesis/Orleans.Streaming.Kinesis.csproj" />
Expand Down
1 change: 1 addition & 0 deletions docs/site/src/content/docs/resources/nuget-packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ These packages don't replace the cluster membership provider.
| [Microsoft.Orleans.Journaling](https://www.nuget.org/packages/Microsoft.Orleans.Journaling) | Pre-release alpha durable journaled collections and values with experimental diagnostic `ORLEANSEXP005`; see the [Journaling overview](../grains/journaling/index.md). |
| [Microsoft.Orleans.Journaling.AzureStorage](https://www.nuget.org/packages/Microsoft.Orleans.Journaling.AzureStorage) | Pre-release alpha Azure Blob and Azure Table providers for Orleans Journaling. |
| [Microsoft.Orleans.Journaling.Redis](https://www.nuget.org/packages/Microsoft.Orleans.Journaling.Redis) | Pre-release alpha Redis provider for Orleans Journaling. |
| [Microsoft.Orleans.Journaling.S3](https://www.nuget.org/packages/Microsoft.Orleans.Journaling.S3) | Pre-release alpha Amazon S3 Express One Zone provider for Orleans Journaling. |
| [Microsoft.Orleans.Transactions](https://www.nuget.org/packages/Microsoft.Orleans.Transactions) | Distributed transaction runtime. |
| [Microsoft.Orleans.Transactions.AzureStorage](https://www.nuget.org/packages/Microsoft.Orleans.Transactions.AzureStorage) | Azure Storage transaction state. |
| [Microsoft.Orleans.Transactions.DynamoDB](https://www.nuget.org/packages/Microsoft.Orleans.Transactions.DynamoDB) | Amazon DynamoDB transaction state. |
Expand Down
1 change: 1 addition & 0 deletions docs/site/src/data/external-link-allowlist.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
"https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html": "AWS serves this public CLI configuration page to browsers but rejects the bounded automated HEAD/GET probe with HTTP 403.",
"https://docs.aws.amazon.com/sdk-for-net/v4/developer-guide/creds-assign.html": "AWS serves this public SDK credential-resolution page to browsers but rejects the bounded automated HEAD/GET probe with HTTP 403.",
"https://docs.aws.amazon.com/streams/latest/dev/introduction.html": "AWS serves this public Kinesis overview to browsers but rejects the bounded automated HEAD/GET probe with HTTP 403.",
"https://www.nuget.org/packages/Microsoft.Orleans.Journaling.S3": "The new S3 journaling package is documented but not yet published; remove this entry and its unpublished API-package entry after publication.",
"https://en.wikipedia.org/wiki/Kalman_filter": "Wikipedia serves this public article to browsers but rejects the bounded automated HEAD/GET probe with HTTP 403.",
"https://www.f5.com/company/blog/nginx/nginx-power-of-two-choices-load-balancing-algorithm": "The F5 site serves this canonical NGINX article to browsers but rejects the bounded automated HEAD/GET probe with HTTP 403."
}
Expand Down
4 changes: 3 additions & 1 deletion docs/site/src/data/unpublished-api-packages.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
{
"description": "Generated API assemblies which are not currently published as standalone NuGet packages.",
"packages": {}
"packages": {
"Microsoft.Orleans.Journaling.S3": "The new provider is awaiting its first NuGet publication."
}
}
32 changes: 32 additions & 0 deletions src/AWS/Orleans.Journaling.S3/Orleans.Journaling.S3.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<PackageId>Microsoft.Orleans.Journaling.S3</PackageId>
Comment thread
ReubenBond marked this conversation as resolved.
Resolved
<PackageReadmeFile>README.md</PackageReadmeFile>
<TargetFrameworks>$(DefaultTargetFrameworks)</TargetFrameworks>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<NoWarn>$(NoWarn);ORLEANSEXP005</NoWarn>
<VersionSuffix Condition="$(VersionSuffix) != ''">$(VersionSuffix).alpha.1</VersionSuffix>
<VersionSuffix Condition="$(VersionSuffix) == ''">alpha.1</VersionSuffix>
<!-- This new package has no published baseline. Enable validation after its first release. -->
<EnablePackageValidation>false</EnablePackageValidation>
<OrleansBuildTimeCodeGen>true</OrleansBuildTimeCodeGen>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="AWSSDK.S3" />
</ItemGroup>

<ItemGroup>
<ProjectReference Include="$(SourceRoot)src\Orleans.Journaling\Orleans.Journaling.csproj" />
</ItemGroup>

<ItemGroup>
<InternalsVisibleTo Include="Orleans.Journaling.Tests" />
</ItemGroup>

<ItemGroup>
<None Include="README.md" Pack="true" PackagePath="\" />
</ItemGroup>
</Project>
3 changes: 3 additions & 0 deletions src/AWS/Orleans.Journaling.S3/Properties/AssemblyInfo.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
using System.Diagnostics.CodeAnalysis;

[assembly: Experimental("ORLEANSEXP005")]
9 changes: 9 additions & 0 deletions src/AWS/Orleans.Journaling.S3/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Microsoft Orleans S3 Journaling

`Microsoft.Orleans.Journaling.S3` provides an `IJournalStorage` implementation backed by Amazon S3 Express One Zone directory buckets.

The provider uses S3 Express append writes (`WriteOffsetBytes`) for WAL appends. For local development and tests against S3-compatible emulators such as MinIO, set `UseS3ExpressAppend = false`, `UseConditionalDelete = false`, and `StorageClass = null` to use portable conditional writes and deletes.

Buckets should be created ahead of time for AWS S3 Express One Zone. `CreateBucketIfNotExists` is intended for local emulators.

Metadata updates rewrite the current WAL using a conditional single-object upload. Publish a checkpoint to compact the WAL before updating metadata when the replacement object would exceed S3's 5 GB (5,000,000,000 byte) single-upload limit. Checkpoint snapshots use the same upload limit.
Loading
Loading