Merge branch 'main' into ianwoodard/infreng-211-shard-stream-tracking

Author: IanWoodard

README.md

@@ -29,14 +29,14 @@ style C fill:#e1f5fe
 
 ### Stream Management
 
-Streams are automatically tracked and cleaned up after inactivity (configurable via `CLEANUP_STREAM_IDLE_SEC`, default 300s).
+Streams are automatically tracked and cleaned up after inactivity (configurable via `CLEANUP_STREAM_IDLE_SEC`, default 120s).
 This prevents memory leaks from:
 
 - Crashed or disconnected publishers
 - Streams that never reach Phase::End
 - Network failures during publishing
 
-A cleanup worker runs periodically (configurable via `CLEANUP_WORKER_INTERVAL_SEC`, default 300s), deleting streams that haven't received any publishes within the inactivity threshold. Active streams (receiving regular publishes) are kept alive indefinitely, supporting long-running or continuous streaming use cases.
+A cleanup worker runs periodically (configurable via `CLEANUP_WORKER_INTERVAL_SEC`, default 120s), deleting streams that haven't received any publishes within the inactivity threshold. Active streams (receiving regular publishes) are kept alive indefinitely, supporting long-running or continuous streaming use cases.
 
 **Note:** While streams themselves are unbounded in duration, client connections (on the Gateway service) may have separate timeout limits. This allows clients to reconnect to ongoing streams as needed.
 
@@ -76,12 +76,20 @@ The `payload` field contains your application data and is typically used with DE
 
 **Size Limits:**
 
-- Maximum message size: 32KB (entire PublishRequest protobuf)
+- Maximum message size: 16KB (entire PublishRequest protobuf)
 - Messages exceeding this limit receive a 413 Payload Too Large response
 - For larger data, split into multiple DELTA messages
 
 The payload must be a valid JSON-like structure in the form of a `google/protobuf/struct.proto`.
 
+**Rate Limits:**
+
+- Maximum 20 requests per second per channel
+- Exceeding returns `429 Too Many Requests` with `Retry-After` header
+- High-frequency publishers should batch messages
+
+**Retention:** Streams hold up to approximately 1200 messages. At maximum publish rate, this provides ~60 seconds of history to consumers to catch up after brief disconnections.
+
 #### Example: Streaming "hello, world!"
 
 ```http

services/publish/README.md

@@ -8,7 +8,7 @@ Each publish updates a stream's activity timestamp in Redis. This allows automat
 
 ### Cleanup Worker
 
-A background worker runs periodically (configurable via `CLEANUP_WORKER_INTERVAL_SEC`, default 300s) and deletes streams with no activity for a configurable duration (via `CLEANUP_STREAM_IDLE_SEC`, default 300s).
+A background worker runs periodically (configurable via `CLEANUP_WORKER_INTERVAL_SEC`, default 120s) and deletes streams with no activity for a configurable duration (via `CLEANUP_STREAM_IDLE_SEC`, default 120s).
 
 ### Phase::End Behavior
 
@@ -22,18 +22,34 @@ This prevents memory leaks from crashed clients or incomplete streams while allo
 
 ## API Limits
 
+### Rate Limiting
+
+Publishers are limited to 20 requests per second using a fixed-window counter in Redis.
+
+- Key: `rate_limit:channel:{org_id}:{channel_id}`
+- Window: 1 second
+- Enforced via a Lua script for atomicity
+
+Rate limiting runs before tracking/publishing to avoid wasted work. The `Retry-After` header tells clients when to retry.
+
+Rate limiting fails closed intentionally since if Redis is having issues with rate limits, it likely isn't going to be able to handle the streams.
+
+**Relationship to stream size:**
+
+At 20/sec with 1200 message streams, consumers have ~60 seconds to recover from disconnections.
+
 ### Message Size
 
-Publish requests are limited to 32KB (configurable via `MAX_MESSAGE_SIZE_BYTES`).
+Publish requests are limited to 16KB.
 Requests exceeding this limit are rejected with `413 Payload Too Large`.
 
-Combined with the stream length limit of 500 messages, this bounds maximum stream size to approximately 16MB per stream.
+Combined with the stream length limit of 1200 messages, this bounds maximum stream size to approximately 19.2MB per stream.
 
 Publishers handling large data should chunk it into multiple DELTA messages within the START/DELTA/END streaming pattern.
 
 ### Stream Length
 
-Streams are automatically trimmed to approximately 500 messages (configurable via `MAX_STREAM_LEN`). Older messages are removed as new ones arrive.
+Streams are automatically trimmed to approximately 1200 messages. Older messages are removed as new ones arrive.
 
 ## Design Decisions
 
@@ -63,3 +79,5 @@ Both `CLEANUP_WORKER_INTERVAL_SEC` and `CLEANUP_STREAM_IDLE_SEC` can be tuned in
 ### Known Edge Cases
 
 - **Sorted sets bloat**: If untrack operations consistently fail, the sorted sets accumulate entries for already deleted streams. The worker will attempt to delete non-existent streams (harmless) but the sorted sets grow. If this becomes a problem, we can add a periodic SCAN to remove ghost entries.
+
+- **Rate Limit Leak**: If EXPIRE fails after INCR succeeds, the rate limit key persists without a TTL. This is unlikely but not impossible.