Skip to content

Optimize Indexes#656

Merged
kraftp merged 14 commits into
mainfrom
kraftp/indexes
May 4, 2026
Merged

Optimize Indexes#656
kraftp merged 14 commits into
mainfrom
kraftp/indexes

Conversation

@kraftp

@kraftp kraftp commented Apr 30, 2026

Copy link
Copy Markdown
Member

Optimize workflow_status table indexes to reduce the cost of maintaining indexes at scale.

  • Indexes on parent_workflow_id, forked_from, and deduplication_id are made partial on those fields not being NULL so they're not maintained when not needed.
  • The index on executor_id is dropped as use in memory map to enforce worker concurrency #644 eliminates the need for it.
  • The index on the status field is split into two partial indexes: one on PENDING workflows for use in recovery; the other on failed workflows (ERROR, CANCELLED, MAX_RECOVERY_ATTEMPTS_EXCEEDED) for rapid troubleshooting. This reduces the number of times the index is updated during a typical workflow lifetime (ENQUEUED->PENDING->SUCCESS).
  • The index on queue_name and status used by the main dequeue query is optimized to index on ("queue_name", "status", "priority", "created_at"), the exact predicates and sort order of the dequeue query. It is now partial on ENQUEUED or PENDING workflows as only the dequeue query needs it.
  • A new partial index is created for rate-limited queues to quickly find recent queries in that queue. That index is only maintained for rate-limited queues (using a new rate_limited field).
  • All migrations use CREATE INDEX CONCURRENTLY for safety when running against production workloads.

@kraftp kraftp marked this pull request as draft April 30, 2026 19:40
@kraftp kraftp marked this pull request as ready for review May 1, 2026 00:03
Comment thread dbos/_schemas/system_database.py
Comment thread dbos/_migration.py
@kraftp kraftp merged commit 91799bb into main May 4, 2026
19 of 20 checks passed
@kraftp kraftp deleted the kraftp/indexes branch May 4, 2026 17:10
kraftp added a commit to dbos-inc/dbos-transact-ts that referenced this pull request May 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants