S3 Compatibility for NATS Object Store

[wpnpeiris]

Proposed change

Introduce a new module to NATS platform, which acts as a gateway for NATS object store.
The new module, (let’s call it nats-s3-gateway, or nats-s3) will be a NATS client (JetStream client) but expose a REST API that adheres to the AWS S3 protocol.
nats-s3 will be its own service, which abstracts away integration with NATS object store and provides S3 protocol implementation.
See the PoC implementation here.

Use case

The S3 API is kind of a de facto standard for accessing Object Storage systems. Many object storage systems such as MinIO, SeaweedFS, JuiceFS, etc support S3 protocol. Having S3 protocol support NATS object store enables integration over HTTP.

Contribution

Yes, we are intending and interested in contributing to NATS.
See a reference implementation here.

[ripienaar]

We are currently working on a HTTP gateway for various aspects of NATS and for sure S3 compatible API would be a great addition.

Really great that you did a POC of this, we'll take a look when we get to thinking about the object store aspect of a above

[kitobelix]

Hey wpnpeiris! Why not also using nats (jetstream) as metadata store?

[rducom]

Considering the scalability and bandwidth capabilities of S3, would it be possible to leverage S3 not only as an API but as a backend storage solution for NATS ?

This approach could offer two operational modes: 1) synchronously storing topics entirely on S3, and 2) using S3 as asynchronous cold storage for long-term or infinite topic retention, maintaining a local hot cache.

Adopting S3 as a primary backend could significantly enhance scalability compared to traditional storage and potentially render NATS nearly stateless (without local disk persistence), as seen with similar projects like Warpstream's adaptation of Kafka.

[derekcollison]

We are looking into many options, including that one. Most folks struggle with the latency, at least in the NATS.io ecosystem. They expect very fast response times both for ingress and egress from JetStream.

[kruegernet]

@derekcollison @rducom this thread is music to my ears. Would tiered storage/prefetch of disk/s3 in the ClickHouse or Apache Druid kind of style be in any way feasible within NATS storage subsystem and with your design philosophy?

[derekcollison]

Possibly.. We have some thoughts..

[kruegernet]

@derekcollison anything being thought about in the open / ADRs or is this kind of Synadia internal conversations to this point? I ask because I have interest in contributing on the OSS here if there is an opportunity.

[derekcollison]

Internal at this point..

[dyumin]

It may end up becoming a good alternative to warpstream, very promising.

[derekcollison]

@dyumin Not sure I would think of it that way. But we are looking at options for cheaper long term storage. For intra AZ costs, those can be mitigated already with NATS to some degree.

[Material-Scientist]

There are S3-API compatible storage solutions that cache data and sync writebacks, which reduces intra AZ costs (and egress in general).

E.g.: