[Feature Request] Session support

[schickling]

Would be great if Sessions similar to the Go SDK were supported in TS as well.

[lorensr]

Here's an alternative:

https://github.com/temporalio/samples-typescript/tree/main/activities-sticky-queues

If anyone has a use case in which this doesn't work well, please post a comment, thanks!

[bergundy]

Note that the main difference between sticky activities and sessions is concurrency control, max concurrent sessions vs to max concurrent activities per worker.

[morgante]

I took a look at the sticky queue but it's not clear that it matches exactly what I need. In particular, I have two doubts/questions:

1. I want the worker to be able to listen on multiple queues. The workflow should only be coupled to a particular worker after it has picked up the first task.
2. If the worker crashes midway through execution, how do I ensure the next worker that spawns for that workflow will start over from the beginning?

[bergundy]

Sorry for the late response.

1. I want the worker to be able to listen on multiple queues. The workflow should only be coupled to a particular worker after it has picked up the first task.

A worker only listens on a single task queue but you can start multiple workers in the same process.
Coupling the workflow to a worker is shown in the sample posted above.

2. If the worker crashes midway through execution, how do I ensure the next worker that spawns for that workflow will start over from the beginning?

You'll need to use schedule to start timeout to detect that the worker went away and restart the sequence of activities.
This is all part of the sticky activity queues sample.

[cretz]

This appears to have been transferred from the TypeScript SDK issue tracker, but for those wondering, all SDKs support a concept of "worker-specific task queues" which basically provides session behavior.

[jinvillaz]

Response: Worker-Specific Task Queue Pattern Implementation

Thank you for the detailed response and for tracking this in #562. I appreciate the context about the redesign requirements and the challenges with the current Go Sessions implementation. I'd like to share our architectural approach to the Worker-Specific Task Queue pattern, addressing the points you raised, as it might provide valuable insights for the Sessions redesign.

Our Architecture Overview

Two-Task-Queue Design

We operate with two separate task queues:

1. Schedule Task Queue (schedule-tq): Dedicated exclusively for schedule workflows that trigger ETL workflows
2. ETL Task Queue (etl-process-worker-{N}): Worker-specific queues for actual ETL workflow execution

This separation ensures that schedule triggers don't interfere with ETL execution, and each worker has its own dedicated queue for ETL workflows.

Sequential Execution Model

Our ETL workflows are sequential by design - activities execute one after another, not in parallel. Therefore:

• maxConcurrentWorkflowTaskExecutions = maxConcurrentActivityTaskExecutions = maxCapacity
• All three values are set to the same number (e.g., 5)
• This ensures that a worker handling 5 workflows can execute 5 workflow tasks and 5 activity tasks concurrently, which matches our sequential model

Key Point: The critical mechanism for our use case is maxCapacity - this is what we use to distribute complete workflows across workers and prevent overloading any single worker.

Addressing Your Points

1. "It is possible to achieve almost the same behavior with the Worker Specific Task Queue pattern, but it is quite daring to properly implement by users"

Our Architectural Approach:

We've implemented a database-backed worker registration and load balancing system that addresses the "daring" aspects:

Worker Registration Layer:

• Each worker instance registers itself in PostgreSQL with a unique task queue identifier (e.g., etl-process-worker-1, etl-process-worker-2)
• Registration includes: status (ACTIVE/INACTIVE), maxCapacity (concurrent workflows), running workflows list, and last heartbeat timestamp
• Workers automatically register on startup and deregister on shutdown

Centralized Load Balancer:

• Before starting any workflow, a centralized service queries the database for available workers
• Uses PostgreSQL advisory locks (pg_advisory_xact_lock) to ensure atomic worker selection and reservation
• Implements least-loaded-first selection algorithm based on maxCapacity
• Atomically reserves a workflow slot before workflow execution begins

Why This Architecture:
The "daring" part is ensuring atomicity and preventing race conditions. By using database transactions with advisory locks, we ensure that:

• Only one workflow can reserve a worker slot at a time
• Worker capacity is checked and updated atomically
• No double-assignment can occur even under concurrent load

2. "User-land implementations have limited ability for controlling execution concurrency at the worker level"

Our Concurrency Control:

Since our workflows are sequential, we set:

• maxConcurrentWorkflowTaskExecutions = maxConcurrentActivityTaskExecutions = maxCapacity

The Critical Mechanism: maxCapacity

maxCapacity is the key to our solution. It represents how many complete workflows a worker can handle simultaneously. This is what we use to:

• Distribute workflows evenly across available workers before they start
• Prevent overloading any single worker with too many workflows
• Balance load by selecting the least-loaded worker (based on current workflow count vs. maxCapacity)

Why This Matters:
Without application-level maxCapacity tracking, Temporal's native concurrency controls would still allow multiple workflows to be assigned to the same worker. Our database-backed maxCapacity ensures that:

• Workflows are distributed intelligently before execution
• No worker gets more workflows than it can handle
• We have visibility into which workflows are running on which workers

3. "Things have significantly improved with Custom Slot Providers and Poller Behaviors"

We haven't explored Custom Slot Providers and Poller Behaviors because they address a different concern than our use case. These features improve concurrency control and polling efficiency within a single worker instance, while our challenge is:

• Distributing workflows across multiple workers before they start
• Ensuring workflow affinity to a specific worker (all tasks execute on the same worker)
• Balancing load at the workflow level across workers

For our specific use case (workflow-level distribution and affinity), our implementation using worker-specific task queues addresses these requirements effectively.

4. "The current design can't offer the kind of DX and operational guarantees that Temporal users expect, notably when it comes to error-handling scenarios"

Error Handling Challenges We've Encountered:

Our implementation revealed several critical error-handling complexities:

Challenge 1: Concurrent Schedule Triggers and Load Balancing

Problem: When multiple schedules trigger simultaneously (e.g., multiple ETL schedules fire at the same time), we need to ensure:

• Each workflow gets assigned to exactly one worker (1:1 assignment)
• Load is balanced across workers
• No race conditions occur where multiple workflows are assigned to the same worker slot

Our Solution - Two-Level Serialization:

1. Application-Level Queue Lock: We use a promise-based queue (queueLock) in the WorkflowBalancerService that serializes all worker reservation requests. This ensures that even if multiple schedule workflows call startEtlWorkflow() simultaneously, they are processed one at a time.

2. Database-Level Atomic Reservation: Within each reservation attempt, we use PostgreSQL advisory locks (pg_advisory_xact_lock) combined with a database transaction to ensure atomicity:

   • Lock is acquired
   • Available workers are queried
   • Least-loaded worker is selected
   • Workflow is added to worker's runningWorkflows array
   • Transaction commits (releasing lock)

Why Both Levels:

• Application-level queue prevents overwhelming the database with concurrent requests
• Database-level lock ensures atomicity even if multiple services/instances try to reserve simultaneously
• Together, they guarantee 1:1 assignment and proper load balancing

Challenge 2: Tracking Workflow Completion and Cleanup

Problem: When a workflow completes (successfully or with failure), we need to:

• Remove it from the worker's runningWorkflows array
• Free up the capacity slot for new workflows
• Handle cases where workflows terminate unexpectedly

Our Solution - Periodic Cleanup:

Each worker runs a periodic cleanup process (every heartbeat interval, default 30 seconds) that:

1. Queries Temporal API to check the status of all tracked workflows
2. Removes workflows that are no longer RUNNING (completed, failed, terminated, or not found)
3. Updates the database to free up capacity

Challenges:

• Cleanup runs on an interval, so there's a delay before capacity is freed up
• Requires querying Temporal API for each tracked workflow, which can be expensive
• Need to handle cases where Temporal API is temporarily unavailable

Challenge 3: Worker Health Monitoring (Heartbeat)

Problem: We need to detect when a worker dies or becomes unresponsive, so we can:

• Mark it as INACTIVE
• Prevent new workflows from being assigned to it
• Clean up stale workflow assignments

Our Solution - Heartbeat Mechanism:

1. Heartbeat Sending: Each worker sends a heartbeat to the database every 30 seconds (configurable), updating its lastHeartbeat timestamp
2. Dead Worker Detection: A cleanup process checks for workers whose lastHeartbeat is older than 2x the heartbeat interval (60 seconds default)
3. Worker Deactivation: Dead workers are marked as INACTIVE
4. Stale Workflow Cleanup: Workflows assigned to dead workers are removed from tracking

Challenges:

• There's a window (up to 60 seconds) where a dead worker might still be considered ACTIVE
• Network issues might cause false positives (worker is alive but can't reach database)
• Need to handle database unavailability gracefully

Use Case: File Download and Sequential Processing

Scenario: Our ETL workflows download large files (often several GB) from vendor APIs and process them sequentially in chunks.

Workflow Pattern:

1. Download file once per workflow
2. Read file in sequential chunks (chunk 1, then chunk 2, then chunk 3, etc.)
3. Process each chunk with stateful operations
4. Maintain file handles and state throughout the workflow

Why Sticky Execution is Critical:

• File handles are tied to worker filesystem: Once a file is downloaded on Worker A, subsequent chunk reads must happen on the same worker to access the local file handle
• State preservation: Partial processing state (buffers, cursors, file positions) is maintained in memory on the worker
• Network efficiency: Re-downloading the file for each activity defeats the purpose of chunked processing
• Resource management: File system resources are tied to specific worker instances

How Our Architecture Addresses This:

• Workflow is assigned to a specific worker (via maxCapacity-based selection) before it starts
• All workflow tasks execute on that worker's specific task queue
• All activities execute on the same worker
• File handles persist throughout workflow execution
• State is maintained in worker memory throughout the workflow lifecycle

Architecture Components

1. Worker Registration Service

• Registers workers in database on startup with unique task queue name
• Updates heartbeats periodically
• Deregisters on shutdown
• Manages worker lifecycle

2. Heartbeat Service

• Sends periodic heartbeats (configurable interval, default 30s)
• Detects and marks dead workers as INACTIVE (no heartbeat for 2x interval)
• Triggers cleanup of stale workflow assignments

3. Workflow Balancer Service

• Application-level queue: Serializes worker reservation requests to prevent concurrent conflicts
• Worker selection: Queries database, filters by capacity, selects least-loaded worker
• Atomic reservation: Uses PostgreSQL advisory locks to ensure 1:1 assignment
• Workflow execution: Starts workflow on selected worker's specific task queue
• Status checking: Verifies if workflow is already running before assignment

4. Database Schema

• Workers table: task_queue (unique), status, max_capacity, running_workflows[], last_heartbeat
• Provides single source of truth for worker state
• Enables atomic operations with advisory locks

Data Flow

Schedule Workflow Triggers (Multiple Simultaneous)
    ↓
Application-Level Queue (Serializes Requests)
    ↓
For Each Request:
    ↓
    Check if Workflow Already Running
    ↓
    Database Query: Find Available Workers
    ↓
    PostgreSQL Advisory Lock Acquired
    ↓
    Filter by Capacity (runningWorkflows.length < maxCapacity)
    ↓
    Select Least-Loaded Worker
    ↓
    Atomically Reserve Slot (Add workflowId to runningWorkflows)
    ↓
    Lock Released
    ↓
    Start Workflow on Worker's Specific Task Queue
    ↓
    All Workflow Tasks Execute on That Worker
    ↓
Periodic Cleanup:
    ↓
    Check Workflow Status via Temporal API
    ↓
    Remove Non-Running Workflows from Tracking
    ↓
    Free Up Capacity

What We've Learned

What Works Well:

• Database-backed worker registration provides reliable state management
• Two-level serialization (application queue + database locks) prevents race conditions
• maxCapacity-based load balancing distributes workflows evenly
• Heartbeat mechanism effectively detects dead workers
• Atomic reservations guarantee 1:1 assignment

Pain Points:

• External database dependency adds operational complexity
• Cleanup delays (30s interval) can temporarily block capacity
• Requires querying the Temporal API for each tracked workflow during cleanup
• Heartbeat window (60s) means dead workers might still receive assignments briefly
• No native integration with Temporal's retry/timeout mechanisms
• Limited observability compared to native Temporal features

Thank you for considering this feedback @mjameswh, and I look forward to contributing to the discussion.