feat: implement alias-first architecture for zero-downtime reindexing

[kapral18]

Summary

After thorough discussion, we need to move from an index-first to an alias-first architecture to support zero-downtime reindexing for shared cluster operators.

Problem Statement

We have two usage modes:

1. Shared cluster operator mode (e.g., @simianhacker's cluster, new OBLT shared cluster with @alejandro.colomina)
2. Personal cluster operator mode

Shared cluster operators need zero-downtime reindexing. When they update indexer logic or mappings, users shouldn't have to wait for reindexing to complete. Currently, with index-first approach, reindexing causes downtime.

Current Architecture (Index-First)

• Index name is primary: kibana (concrete index)
• Alias is secondary: kibana-repo → kibana (for MCP discovery)
• npm run index -- repo:kibana → creates/uses index kibana
• Index names are stable and user-facing

Limitations:

• No zero-downtime reindexing
• Users see actual index names in list_indices
• Can't hot-swap indices without downtime

New Architecture (Alias-First)

• Alias is primary: kibana (public-facing identifier)
• Index is ephemeral: kibana-{timestamp} or kibana-v{N} (internal implementation detail)
• npm run index -- repo:kibana → creates kibana-{timestamp}, indexes to it, sets alias kibana → kibana-{timestamp}
• All references become alias references
• list_indices shows aliases, not indices

Benefits:

• ✅ Zero-downtime reindexing via alias swapping
• ✅ Clean public API (aliases only)
• ✅ Works for both shared and personal cluster operators
• ✅ Hot-swappable indices
• ✅ No -repo suffix requirement (users can pass any alias name)

Operational Model (Stateless & Efficient)

To support Stateless CronJobs on immutable infrastructure (Kubernetes/GitOps) while ensuring Resource Efficiency, we separate the operational concerns:

1. Steady State (CronJob):

   • Command: npm run index -- kibana
   • Role: Incremental updates.
   • Resources: Lightweight (Low CPU/RAM).
   • Behavior: Resolves kibana alias to current index, indexes diffs.

2. Maintenance State (One-Off Job):

   • Command: npm run index -- kibana --clean
   • Role: Full Reindex & Alias Swap.
   • Resources: Heavyweight (High CPU/RAM).
   • Behavior: Creates new index, full index, atomic swap, delete old.

This separation prevents the "Super CronJob" anti-pattern (sizing a cronjob for peak 16h reindex load) and keeps the schedule pure.

Concurrency Control (GitOps Compatible)

To avoid infrastructure mutation (e.g. kubectl patch suspend which causes GitOps drift) and overlapping runs:

• Repository-Scoped Locking: The application uses a lock in the settings index (e.g. reindex_in_progress_kibana).
• Maintenance Job: Acquires lock → Reindexes → Releases lock.
• CronJob: Checks lock on startup. If locked, skips run (exits 0).

This allows the CronJob to remain "running" (according to K8s) but "paused" (logically) during maintenance, satisfying immutable infrastructure constraints.

Required Changes

1. Index Name Generation

• Auto-generate versioned index names: {aliasName}-{timestamp} or {aliasName}-v{N}
• Index names become ephemeral/internal

2. Alias Management

• Create/update aliases as the primary operation
• All index parameters become alias references
• Alias becomes the stable identifier

3. Old Index Cleanup

• Track previous index versions
• By default, cleanup old indices after reindexing
• Add --keep-old-indices flag for operators who want to keep old indices temporarily

4. API Changes

• npm run index -- repo:kibana → kibana is now an alias, not an index name
• All internal references switch from index names to aliases
• Settings index needs to track alias → version mapping

5. MCP Integration

• list_indices shows aliases, not indices
• MCP already uses aliases for discovery, so this part works

6. Interface Changes

• Remove -repo suffix requirement
• User can pass whatever alias name they want: npm run index -- repo:my-alias
• Alias name becomes the public-facing identifier

Implementation Details

Example Flow

User Command:
  npm run index -- kibana

Execution:
  1. Generate index name: `kibana-1733856000000` (timestamp)
  2. Create index: `kibana-1733856000000`
  3. Index all data to: `kibana-1733856000000`
  4. Create/update alias: `kibana` → `kibana-1733856000000`
  5. If old index exists: Delete old index (unless `--keep-old-indices` flag)
  6. MCP discovers via alias: `kibana`
  7. `list_indices` shows: `kibana` (alias), not `kibana-1733856000000` (index)

Zero-Downtime Reindexing Flow (with Locking)

Scenario: Operator wants to reindex with new mappings via Maintenance Job

1. Maintenance Job Starts (`--clean`)
2. Acquires lock: `reindex_in_progress_kibana = true`
3. Current state: `kibana` (alias) → `kibana-1733856000000` (index)
4. Create new index: `kibana-1733856100000` (new timestamp)
5. Reindex all data to: `kibana-1733856100000`
6. (Meanwhile) CronJob wakes up → Sees lock → Skips run
7. When complete, atomically swap: `kibana` → `kibana-1733856100000`
8. Delete old index: `kibana-1733856000000`
9. Release lock: `reindex_in_progress_kibana = false`
10. Users/CronJob continue via `kibana` alias (zero downtime!)

Trade-offs

• Drawback: On every reindex, users will see a new index in index_management dashboard with changed timestamp
• Benefit: This is acceptable for the zero-downtime capability it provides
• Benefit: Users/agents only interact with clean alias names via list_indices

Related

• Supersedes Automatically create -repo alias when indexing repositories #130 and feat: automatically create -repo alias when indexing (with index name normalization) #131
• Enables zero-downtime reindexing for shared cluster operators
• Complements MCP server dual discovery strategy

[Coolomina]

2. Create new index: `kibana-1733856100000` (new timestamp)
3. Reindex all data to: `kibana-1733856100000`

Would we be running a full re-index every time npm run index is executed?, what would be the incremental strategy in this proposal? First clone, then incremental on the new index, then hot swap?

[kapral18]

Currently

npm run index is a unified command now. It detects the presence of index with the same name. If it's there it changes to incremental mode and just sends whatever is missing from current .repos. So after the very first run all subsequent runs are by default in incremental mode for the same index.

we have --pull and --clean to control what we want. Former will git pull, the latter will fully reindex.

With the new approach

npm run index will still be unified but now we will operate with index through alias. When you rerun after creation it will still be in incremental mode, we will only updated that one generated index. Unless you pass --clean. In which case we will generate a completely new generated index and relink it to the alias you provided with npm run index -- aliasname? Makes sense?

So from users usage perspective nothing changes. Everything works the same, except for what they previously thought was index name is now alias name.

[Coolomina]

Unless you pass --clean. In which case we will generate a completely new generated index and relink it to the alias you provided with npm run index -- aliasname

If I understand the suggested approach correctly, arbitrarily passing new args to a CLI won't work very well in an immutable infrastructure environment. Let's consider this scenario with such approach:

1. The developer deploys a cron job that runs node dist/index.js index my-repo on a specific schedule. The index performs a complete index once, then runs incremental updates from that point onward.
2. For whatever reason, the developer needs to fully re-index the repository. Then, following this issue's suggestion, they would need to somehow instruct the cron job to run node dist/index.js index my-repo --clean instead of the original command, which would orchestrate the hotswapping of the indices as you describe. If left in this state, the cron job would fully re-index the repository on every schedule it runs, which is highly suboptimal.
3. To save resources, the developer decides to revert back to the original command node dist/index.js index my-repo so incrementals are run on next schedules.

Returning to my point, in an immutable infrastructure scenario (such as a Kubernetes or ECS Fargate environment), each change to the executing command (two in the previous scenario) requires a redeployment of the container infrastructure, which must also be orchestrated. This will, in time, be an operational burden for whoever operates the infrastructure where the cron jobs are deployed.

Proposal

I propose to delegate the state of needing to reindex to the _settings index: a new field needsFullReindex that defaults to false and instructs the indexer component whether a clean run needs to be carried out.

The main dependency for this implementation is that a new command must be created. Let's imagine node dist/index.js index:needs-reindex my-repo. When invoked, it would set needsFullReindex field from the _settings index to true.

In comparison to the previous flow, this one would be:

1. The developer deploys a cron job that runs node dist/index.js index my-repo on a specific schedule. The index performs a complete index once, then runs incremental updates from that point onward.
2. For whatever reason, the developer needs to fully re-index the repository. They'd somehow run node dist/index.js index:needs-reindex my-repo. This would set needsFullReindex to true.
3. Next scheduled run will consume needsFullReindex, given that it's true, it'll reindex with all the orchestration described in the issue. When finished, it must set needsFullReindex to false.
4. Subsequent runs will consume needsFullReindex, see it's false, and run an incremental without modifying any infrastructure configuration.

Thoughts, folks?

[kapral18]

@Coolomina respectfully we are speaking about an occasional maintenance event. It's absolutely normal to stop cron, run a maintenance job and restart cron when it's done. This is exactly that. Normal operation mode for a cronjob would be npm run index -- kibana which will be incremental. And when we need to reindex fully you run a one-time job with npm run index -- kiban --clean and then restart cron as is. Everything will work smoothly without introducing unnecessary complexity and statefulness 99% of the time to avoid 1 action. Of course initially since we do heavy development the need to rerun cron job is increased but overtime it will get less and less.

[kapral18]

@Coolomina

I mean can you share what prevents us from from handling this in Ops layer instead of changing application layer to accomodate that. For ex. using a shell script that goes over a list of repos to reindex and for each:

1. kubectl patch cronjob <name> -p '{"spec": {"suspend": true}}'
2. kubectl create job maintenance-<name> --from=cronjob/<name> -- npm run index -- kibana --clean --concurrency 100
3. kubectl wait --for=condition=complete job/maintenance-<name> --timeout=20h
4. kubectl patch cronjob $REPO -p '{"spec": {"suspend": false}}'

[Coolomina]

@Coolomina

I mean can you share what prevents us from from handling this in Ops layer instead of changing application layer to accomodate that. For ex. using a shell script that goes over a list of repos to reindex and for each:

1. kubectl patch cronjob <name> -p '{"spec": {"suspend": true}}'
2. kubectl create job maintenance-<name> --from=cronjob/<name> -- npm run index -- kibana --clean --concurrency 100
3. kubectl wait --for=condition=complete job/maintenance-<name> --timeout=20h
4. kubectl patch cronjob $REPO -p '{"spec": {"suspend": false}}'

A deployment, which can happen at any time, will override all of those options, therefore reverting it to the original state and invalidating the scripts.

Let me elaborate on what Immutable infrastructure means and why running ad hoc mutations on the cluster state is not the best idea.

Immutable infrastructure in plain terms

In an immutable setup, which is the gold standard in infrastructure management:

• You don’t change live systems by hand
• The desired state lives in code (Git)
• The cluster continuously reconciles to match that code (via GitOps / controllers)
• If something needs to change, you change the source of truth, not the running system

Kubernetes is designed around this idea. Anything you change imperatively with kubectl is, by definition, out-of-band.

What the proposed scripts do

• Temporarily mutate live cluster state
• Create ad-hoc workloads that are not declared anywhere
• Assume nothing else is touching those resources at the same time
• Rely on humans/scripts to clean up and restore the state correctly

This works in a mutable ops model (say, a PoC VM where you're running the indexer), but it breaks down in an immutable one.

Why it's not the best idea

It fights the source of truth

If CronJobs are defined in Git (Helm, Kustomize, Terraform, etc.):

• kubectl patch changes the live object
• Git still says suspend: false
• A reconciler (ArgoCD / Flux / Terraform) will eventually say: “This does not match desired state”, and flip it back while the scripts are running
• Result: flaky, non-deterministic behavior.

State drift becomes invisible and unreviewable

Changes to the state won't go through code review and aren't auditable in git history, or anywhere.

Failure handling is fragile and manual

If the job fails halfway, the cluster restarts, the script times out or kubectl wait exits early, we now have to manually inspect and repair state.

tl; dr

I'm against mutating infrastructure; IMO this is porting a missing indexer feature into infrastructure glue. If the team thinks it's still the preferred approach, I won't obviously block it. We'd need to be sure deployments won't happen when a --clean reindex is happening.

Sorry for the wall of text, hope it was somehow useful to read for the team!

[kapral18]

@Coolomina thanks

gemini-3-pro has a better compromise solution :D

[simianhacker]

For the index naming, I'm thinking we should use the package version and get into the habit of updating the package version when the schema changes. Almost all of the re-indexing I've done over the past 6 months (if not all) has been due to schema changes. The unix timestamp is a little hard to consume for a human and makes debugging error prone.

I like the idea of using the *_settings index for more than just the version.

[kapral18]

@simianhacker that use case is valid but that's not the only times you CAN use reindex. I can use reindex any time I want if I pass --clean and local use case is as important as CI one that you guys use. Also you might want to reindex because of some unrecoverable error or corruption that happened in the middle of indexing in your env(local or CI). And you have to produce a new unique temp index to maintain zero downtime. And we can't use one naming for CI use case and another naming for all other use cases, the naming imho should be unified to avoid even more complexity.

I think it's ok to include schema version in the name but we still need to have a unique dynamic part that's per re-index call
ex. kibana-scsi-schema-v8.19.1-ef4ab5 or maybe even just kibana-<hash> and put scsi-schema-version: as a field in *_settings file to keep track

And also to be able to use package version we need wire up a release process with semantic versioning and github releases and tags as a grown up project :) I can do that but then it shifts this ticket a little further.

[kapral18]

@simianhacker also on the previous point about release process my 2c is we are not ready yet for grown up versioning. We have too many breaking issues to guarantee backwards compat to anybody at this point. I think we need to do some body of bugfix/enhancement and research work before we can introduce version 1.0.0 and start supporting backwards compat and ship out atomic backwards compatible updates. I would at least wait a little.