Skip to content

Default maxCachedWorkflows calculation doesn't account for VM isolate memory being outside V8 heap #1843

@alonp99

Description

@alonp99

The default maxCachedWorkflows formula uses maxHeapMemory (--max-old-space-size) to calculate
how many workflows to cache. However, workflow VM isolates allocate native memory outside
the V8 heap, not inside it.

This causes unexpected OOMs in containerized environments where:

  1. --max-old-space-size is set high (e.g., 4.6GB) relative to container limit (5Gi)
  2. Temporal calculates maxCachedWorkflows = ~2600
  3. VM isolates consume ~2.6GB of NATIVE memory (not heap)
  4. Total memory = heap + VM cache + overhead > container limit → OOMKill

Suggested improvements:

  1. Documentation: Warn that VM isolates use native memory, not heap memory
  2. Default calculation: Consider using a more conservative default, or factor in
    that VM cache is additive to heap usage
  3. Logging: Log a warning if calculated maxCachedWorkflows * ~1MB + maxHeapMemory
    exceeds available system memory

Environment: TypeScript SDK 1.11.7, Node.js 22, Kubernetes

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions