chore: update documentation

Author: samuelabebayehu

omod/src/main/resources/_etl/derived/follow_up/README.md

@@ -22,8 +22,8 @@ That arrangement had three problems:
 
 This folder solves all three: the table is named `mamba_fact_follow_up` (peer of
 `mamba_fact_client` and `mamba_fact_location_*`), the rebuild is a clean
-DROP+rebuild on every ETL run, and the layout is replicated mechanically for
-the other 8 forms.
+blue-green (staging + atomic rename) rebuild on every ETL run, and the layout
+is replicated mechanically for the other 8 forms.
 
 ## Layout
 
@@ -32,7 +32,6 @@ follow_up/
 ├── README.md                                          this file
 ├── sp_makefile                                        lists sp_data_processing_derived_follow_up.sql
 ├── sp_data_processing_derived_follow_up.sql           entry-point called by sp_mamba_data_processing_etl
-├── sp_compress_follow_up_flat_tables.sql              post-create compression for the 10 source flat tables
 └── facts/
     └── follow_up/
         ├── sp_makefile                                lists the 3 SP files below
@@ -46,77 +45,75 @@ since the materialization is always full-rebuild, there is no separate
 `_update.sql` for delta refresh. (The original `_update.sql` was reverted —
 see "Full rebuild on every run" below.)
 
-## Full rebuild on every run
+## Full rebuild on every run (blue-green)
 
 The entry-point `sp_data_processing_derived_follow_up` performs a full rebuild
-of `mamba_fact_follow_up` on every ETL run:
-
-1. `DROP TABLE IF EXISTS mamba_fact_follow_up` — always, regardless of mode
-2. `CALL sp_fact_follow_up()` — creates the table (with `ROW_FORMAT=COMPRESSED
-   KEY_BLOCK_SIZE=8`) and populates it via `TRUNCATE` + 10-way LEFT JOIN INSERT
-3. `CALL sp_compress_follow_up_flat_tables()` — re-applies compression to the
-   10 source flat tables and the new materialization (idempotent metadata-only
-   change on already-compressed tables)
+of `mamba_fact_follow_up` on every ETL run, using a blue-green (staging +
+atomic swap) strategy so the live table stays available with old data for
+the whole rebuild:
+
+1. `DROP TABLE IF EXISTS mamba_fact_follow_up_staging` — clean up any leftover
+   staging table from a previously aborted run.
+2. `CALL sp_fact_follow_up()` — creates `mamba_fact_follow_up_staging` (with
+   `ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8`) and populates it via `TRUNCATE`
+   + 10-way LEFT JOIN INSERT. The live `mamba_fact_follow_up` is untouched
+   and fully queryable throughout this step.
+3. `DROP TABLE IF EXISTS mamba_fact_follow_up_old` — guard against a
+   pre-existing `_old` table left by a previously crashed run.
+4. Atomic `RENAME TABLE mamba_fact_follow_up TO mamba_fact_follow_up_old,
+   mamba_fact_follow_up_staging TO mamba_fact_follow_up` — readers see either
+   the fully old or fully new table, never a partial rebuild. The archived
+   `_old` table is dropped immediately after.
 
 **Why full rebuild instead of incremental:**
 
 - ETL is user-triggered (after data changes), not a high-frequency timer.
 - The user expects a known-correct state after each run, with no drift and
   no dependence on a `_mamba_etl_schedule` watermark.
-- The performance cost is acceptable: a 10-way JOIN over 5M rows + 10
-  idempotent `ALTER TABLE` statements takes ~5–10 seconds on a typical
-  8 GB / HDD box. Within the headroom of a 5-minute ETL schedule.
-- Compression keeps the disk footprint small and the I/O bounded.
+- The performance cost is acceptable: a 10-way JOIN over 5M rows takes
+  ~5–10 seconds on a typical 8 GB / HDD box. Within the headroom of a
+  5-minute ETL schedule.
+- Blue-green keeps the live table queryable throughout, instead of dropping
+  it up front and leaving a "table not found" gap for the duration of the
+  rebuild.
 
 **Behavior change from the previous incremental design:**
 
 - The `sp_fact_follow_up_update.sql` (delta refresh) was deleted. It is
   no longer called from any SP.
 - The wrapper `sp_fact_follow_up.sql` lost its `etl_incremental_mode`
-  parameter and the IF/ELSE branch. It always calls `_create` then `_insert`.
-- The entry-point `sp_data_processing_derived_follow_up.sql` always does
-  `DROP TABLE IF EXISTS mamba_fact_follow_up` (no mode gating).
+  parameter and the IF/ELSE branch. It always calls `_create` then `_insert`,
+  building into `mamba_fact_follow_up_staging`.
+- The entry-point `sp_data_processing_derived_follow_up.sql` no longer drops
+  the live table up front — it drops leftover staging/`_old` tables, builds
+  staging, then does the atomic rename swap described above.
 - The `etl_incremental_mode` parameter is kept in the entry-point's
   signature for backward compatibility with the call site in
   `sp_mamba_data_processing_etl`, but is intentionally ignored.
 
 ## Compression
 
-Both `mamba_fact_follow_up` and the 10 source flat tables are configured with
-`ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8`. This applies to deployments on both
-small (8 GB / HDD) and large (powerful) machines — the trade is cheap CPU for
-less I/O, which is the right trade on either profile.
-
-### How compression is applied
-
-- **`mamba_fact_follow_up`**: created with `ROW_FORMAT=COMPRESSED
-  KEY_BLOCK_SIZE=8` in `sp_fact_follow_up_create.sql`. Idempotent via
-  `CREATE TABLE IF NOT EXISTS`.
-- **The 10 source flat tables**: the core (separate repo) creates them as
-  default `ROW_FORMAT=DYNAMIC` via `sp_mamba_flat_encounter_table_create`.
-  We can't change that from this repo, so we re-apply compression via
-  `sp_compress_follow_up_flat_tables.sql`, called from the entry-point
-  after the materialization runs. The 10 `ALTER TABLE` statements are
-  idempotent: on an already-compressed table MySQL 8 performs a
-  metadata-only change (no rewrite); on an uncompressed table it rewrites
-  the table once.
+`mamba_fact_follow_up` (built as `mamba_fact_follow_up_staging`, then renamed
+into place) is created with `ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8` in
+`sp_fact_follow_up_create.sql`, idempotent via `CREATE TABLE IF NOT EXISTS`.
+The trade is cheap CPU for less I/O on both small (8 GB / HDD) and large
+deployments.
+
+The 10 source flat tables (`mamba_flat_encounter_follow_up`, `_1`…`_9`) are
+**not** compressed by this repo. They're created by the core
+(`sp_mamba_flat_encounter_table_create`, separate repo) with the default
+`ROW_FORMAT=DYNAMIC`, and we can't change that from here. An earlier version
+of this folder re-applied compression to those 10 tables via
+`sp_compress_follow_up_flat_tables.sql`, called from the entry-point after
+the materialization ran — that SP has been removed; only the materialized
+fact table itself is compressed now.
 
 ### Why `KEY_BLOCK_SIZE=8` and not 4 or 16
 
 - `KEY_BLOCK_SIZE=4`: ~70% compression, ~20% CPU, more risk of compression failures on busy pages
 - `KEY_BLOCK_SIZE=8`: ~55% compression, ~10% CPU — sweet spot for HDD
 - `KEY_BLOCK_SIZE=16`: ~30% compression, ~5% CPU — marginal win over DYNAMIC
 
-### What about existing deployments
-
-This folder assumes a fresh deploy: the next ETL run that hits the
-full-rebuild path will produce compressed tables. For an existing
-deployment with >5 GB of uncompressed data already on disk, run a
-one-time `ALTER TABLE mamba_flat_encounter_follow_up[_1.._9]
-ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8` on each of the 10 flat tables
-during a maintenance window. The compression cost is ~minutes per
-GB on HDD.
-
 ## Known limitations (out of scope)
 
 - **Drift correction** — the full-rebuild model makes this moot. Every

omod/src/main/resources/_etl/derived/follow_up/facts/follow_up/sp_fact_follow_up.sql

@@ -1,9 +1,9 @@
 -- Wrapper for the follow-up materialization.
 -- Always performs a full rebuild: _create (idempotent CREATE TABLE)
--- followed by _insert (TRUNCATE + 10-way LEFT JOIN INSERT).
--- The table is dropped by sp_data_processing_derived_follow_up
--- before this wrapper is called, so the wrapper is responsible only
--- for the create-then-populate sequence.
+-- followed by _insert (TRUNCATE + 10-way LEFT JOIN INSERT), both against
+-- mamba_fact_follow_up_staging. The live mamba_fact_follow_up table is
+-- left untouched by this wrapper; sp_data_processing_derived_follow_up
+-- swaps staging into place with an atomic RENAME after this call returns.
 DELIMITER //
 
 DROP PROCEDURE IF EXISTS sp_fact_follow_up;