Skip to content

Add configurable storage type to NATS streaming provider - #10028

Merged
ReubenBond merged 1 commit into
dotnet:mainfrom
Cluster7ck:feat/nats-streaming-storage-type
Apr 23, 2026
Merged

ReubenBond merged 1 commit into
dotnet:mainfrom
Cluster7ck:feat/nats-streaming-storage-type

Conversation

@Cluster7ck

@Cluster7ck Cluster7ck commented Apr 23, 2026 •

Copy link
Copy Markdown
Contributor

Passes StorageType into StreamConfig so operators can target servers configured with only memoryStore (or only fileStore). Requesting a storage type the server has not provisioned otherwise fails with "insufficient storage resources available" at stream creation, since JetStream reports a 0-byte budget for the disabled backend.

Microsoft Reviewers: Open in CodeFlow

passes StorageType into StreamConfig so operators can target servers
configured with only memoryStore (or only fileStore). Requesting a
storage type the server has not provisioned otherwise fails with
"insufficient storage resources available" at stream creation, since
JetStream reports a 0-byte budget for the disabled backend.
@Cluster7ck

Copy link
Copy Markdown
Contributor Author

@dotnet-policy-service agree

@Cluster7ck

Copy link
Copy Markdown
Contributor Author

@galvesribeiro can you take a look at this? I am aware that this will break horribly if the nats server/cluster falls over, but that seems like a risk you should be able to take. It's the same for redis streaming depending on the server config.

@galvesribeiro

galvesribeiro commented Apr 23, 2026 •

Copy link
Copy Markdown
Member

Sorry for the delay. I see the point of the PR but this is very unsafe. Not sure we should support this.

The correct approach from NATS perspective for this kind of scenarios where persistence is not required is to use NATS Core, not JetStream.

But yeah, if this is wanted, your implementation is correct.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an operator-configurable JetStream stream storage backend selection to the Orleans NATS streaming provider so deployments can target servers provisioned with only file or only memory storage.

Changes:

  • Introduces NatsOptions.StorageType (defaulting to StreamConfigStorage.File) with documentation.
  • Applies StorageType to the JetStream StreamConfig during stream creation/update.
  • Adds unit/integration tests covering the new default and verifying storage type is applied on the created stream.
Show a summary per file
File Description
test/Extensions/Orleans.Streaming.NATS.Tests/NatsOptionsTests.cs Adds tests for default storage type and verifies storage backend is applied in JetStream stream config.
src/Orleans.Streaming.NATS/Providers/NatsConnectionManager.cs Passes configured storage type into StreamConfig when creating/updating the stream.
src/Orleans.Streaming.NATS/NatsOptions.cs Adds StorageType option and documents expected behavior and failure modes.

Copilot's findings

  • Files reviewed: 3/3 changed files
  • Comments generated: 3

Comment on lines +65 to +76
/// <summary>
/// The storage backend used by the NATS JetStream stream.
/// Use <see cref="StreamConfigStorage.File"/> for durability across NATS
/// server restarts, or <see cref="StreamConfigStorage.Memory"/> for lower
/// latency when durability is not required and the NATS server is
/// configured with a memory store. The NATS server must have the
/// corresponding storage type enabled; requesting a storage type the
/// server has not provisioned results in an "insufficient storage
/// resources available" error at stream creation.
/// Defaults to <see cref="StreamConfigStorage.File"/>.
/// </summary>
public StreamConfigStorage StorageType { get; set; } = StreamConfigStorage.File;

Copilot AI Apr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StorageType is user-configurable (via config binding) but NatsStreamOptionsValidator does not validate that the value is a defined StreamConfigStorage enum member. This can result in an invalid numeric value being sent to JetStream and failing at runtime. Consider adding a validator check (e.g., Enum.IsDefined) and throwing an OrleansConfigurationException with guidance on valid values (File/Memory).

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I agree here on adding the validation.

var connectionManager = new NatsConnectionManager(providerName, NullLoggerFactory.Instance, options);
await connectionManager.Initialize();

await using var natsConnection = new NatsConnection();

Copilot AI Apr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test uses new NatsConnection() (default URL) even though NATS availability is determined via NatsTestConstants (which pins Url = nats://127.0.0.1:4222). To avoid localhost/IPv6 resolution mismatches and keep tests consistent with the rest of the NATS test suite, use NatsTestConstants.CreateConnection() (or pass NatsTestConstants.NatsClientOptions) for the verification connection/context.

Suggested change
await using var natsConnection = new NatsConnection();
await using var natsConnection = new NatsConnection(NatsTestConstants.NatsClientOptions);

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also here.

NumReplicas = 1,
PartitionCount = 2,
ProducerCount = 1,
StorageType = storageType

Copilot AI Apr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency with the verification connection and NatsTestConstants settings, consider setting options.NatsClientOptions = NatsTestConstants.NatsClientOptions before constructing NatsConnectionManager. That ensures the manager creates the stream against the same server endpoint that the test later queries.

Suggested change
StorageType = storageType
StorageType = storageType,
NatsClientOptions = NatsTestConstants.NatsClientOptions

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And here.

@galvesribeiro galvesribeiro left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I think the ideal usage for NATS JetStream in production workloads is for having persistence, there may be legit cases to have ephemeral streams in memory, but for those cases, I'd think it would be better to use NATS Core, but this has a different semantics on the subscriptions which would require another implementation of the provider.

Nonetheless, this change is ok. Just minor comments in terms of consistency on the tests.

@ReubenBond
ReubenBond added this pull request to the merge queue Apr 23, 2026
Merged via the queue into dotnet:main with commit b5c347b Apr 23, 2026
66 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators May 24, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants