Skip to content

feat: Add block_swap_optimizer_patch_params option for supporting some optimizers with block swapping#899

Merged
kohya-ss merged 4 commits into
mainfrom
feat-z-image-train-blockswap-compatiblity-with-various-optimizers
Feb 15, 2026
Merged

feat: Add block_swap_optimizer_patch_params option for supporting some optimizers with block swapping#899
kohya-ss merged 4 commits into
mainfrom
feat-z-image-train-blockswap-compatiblity-with-various-optimizers

Conversation

@kohya-ss

@kohya-ss kohya-ss commented Feb 14, 2026

Copy link
Copy Markdown
Owner

This pull request introduces a new option, --block_swap_optimizer_patch_params, to improve compatibility when using block swapping with certain optimizers in low VRAM environments. The documentation (both English and Japanese) has been updated to explain the new option and provide clearer recommendations for various VRAM scenarios. Additionally, the training script has been updated to implement the patch and support the new command-line argument.

Block swapping and optimizer compatibility:

  • Added the --block_swap_optimizer_patch_params option to the training script (zimage_train.py) and its argument parser, allowing gradients to be moved to the same device as parameters during the optimizer step. This patch enables block swapping to work with AdamW and Adafactor optimizers, but not with AdamW8bit or others due to implementation differences. [1] [2]

Documentation updates (English and Japanese):

  • Updated the documentation in docs/zimage.md to describe the new --block_swap_optimizer_patch_params option, its purpose, and its compatibility with different optimizers. [1] [2]
  • Expanded and clarified recommended settings for limited VRAM environments, providing several configuration options and guidance on expected VRAM usage, training speed, and accuracy trade-offs. [1] [2]

@kohya-ss kohya-ss merged commit 3b845bd into main Feb 15, 2026
1 check passed
@kohya-ss kohya-ss deleted the feat-z-image-train-blockswap-compatiblity-with-various-optimizers branch February 15, 2026 04:50
siraxe pushed a commit to siraxe/musubi-tuner that referenced this pull request Mar 6, 2026
@claude
Squashed commit of the following:
8e8752b 
commit 3d7bf14
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 21:21:12 2026 +0300

    fix LTX-2.3 VAE decoder mismatch error (AkaneTendo25#40)

commit eae8c12
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 21:01:11 2026 +0300

    fix: materialize missing VAE buffers for LTX-2.3 compatibility

commit b6fe330
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 20:20:26 2026 +0300

    remove LTX-2 to 2.3 LoRA converter (AkaneTendo25#40)

commit 9ded007
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 19:25:16 2026 +0300

    update docs

commit 7140a71
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 16:11:12 2026 +0300

    add LTX-2.3 metadata keys to full fine-tune and LoRA checkpoint metadata

commit 5be6108
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 15:38:00 2026 +0300

    add LTX-2.3 training infrastructure: version detection, stretched sigma sampler, CLI args, update docs (AkaneTendo25#40)

commit db6d575
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 13:47:00 2026 +0300

    fix: align transformer parity (prompt_adaln routing + cross-modal perturbation skip semantics)

commit 271395d
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 13:18:19 2026 +0300

    (experimental) add LTX-2 -> LTX-2.3 LoRA converter with shape-aware mapping and SVD fallback (AkaneTendo25#40)

commit cf5f673
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 11:28:42 2026 +0300

    (WIP) add LTX-2.3 compatibility: dual text projection, split connectors, cross-attn AdaLN, BWE vocoder (AkaneTendo25#40)

commit 908c40a
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 09:59:32 2026 +0300

    small doc fix

commit 9164339
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 20:39:57 2026 +0300

    enforce include_patterns as strict whitelist

commit c94c173
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 20:39:50 2026 +0300

    add physical audio-only transformer training option

commit f55704e
Merge: 1296186 1a23c66
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 14:20:10 2026 +0300

    Merge branch 'ltx-2' into ltx-2-dev

commit 1a23c66
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 14:19:44 2026 +0300

    fix non-relevant info in docs

commit 1296186
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 00:44:59 2026 +0300

    add opt-in block checkpointing improvements (pinned slab pool, eviction scoring, multi-block prefetch)

commit c5e0ad2
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 3 22:39:51 2026 +0300

    add per-module learning rate groups for LoRA training (AkaneTendo25#1)

commit fd87446
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 3 15:32:38 2026 +0300

    fix: audio channel coercion when neither dimension matches channel count (AkaneTendo25#12)

commit fc9fdbb
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 2 21:30:52 2026 +0300

    fix misleading docs

commit 36d4b35
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 2 13:52:40 2026 +0300

    stabilize audio-only sequence length and auto-detect audio mode

commit 3477108
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 2 11:16:12 2026 +0300

    update generate script

commit 794c9b6
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 22:03:45 2026 +0300

    update docs

commit 02db38a
Merge: 57bef9b 29318b1
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 20:41:48 2026 +0300

    Merge branch 'ltx-2' into ltx-2-dev

    # Conflicts:
    #	src/musubi_tuner/ltx2_train_network.py

commit 29318b1
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 20:38:23 2026 +0300

    add audio_bucket_strategy flag and fix audio-only mode issues (AkaneTendo25#12)

commit 96ae865
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 13:19:11 2026 +0300

    fix CREPA resume crash AkaneTendo25#37

commit 57bef9b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 13:19:11 2026 +0300

    fix CREPA resume crash AkaneTendo25#37

commit ef486aa
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 27 22:47:20 2026 +0300

    make subprocess audio decode default

commit 11526a7
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 27 20:28:07 2026 +0300

    add audio DOP, rename comfy output checkpoint to .comfy.safetensors, update docs

commit 9c373fe
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 27 12:10:15 2026 +0300

    fix audio_loss_balance_min default

commit 7fb3c61
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 27 11:54:55 2026 +0300

    skip FPS resampling for near-matching frame rates, save both original and Comfy LoRA by default

commit e0e1d13
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Feb 26 01:47:55 2026 +0300

    introduce experimental motion-prior replay and EWC regularization pipeline (FFT only)

commit 8cd3394
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 25 16:56:51 2026 +0300

    add --save_checkpoint_metadata for JSON sidecar files alongside checkpoints (AkaneTendo25#32)

commit 6bdd8aa
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 25 16:28:15 2026 +0300

    allow loading Gemma from safetensors (AkaneTendo25#20), change checkpoint cleanup behavior, rework standalone inference script

commit 672c487
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 15:55:33 2026 +0300

    make sample preview failures during FFT non-fatal

commit ecc2baa
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 14:50:26 2026 +0300

    update docs, remove redundant logging

commit 12f1121
Merge: 6205c57 d967597
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 12:13:00 2026 +0300

    Merge commit 'd967597f72aa70b507b97ed7c2aec7e9f315b87e' into ltx-2-dev

    # Conflicts:
    #	docs/ltx_2.md

commit 6205c57
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 11:23:30 2026 +0300

    add ComfyUI-compatible and merged checkpoint saving for full fine-tune, fix sampling and metadata errors

commit dc30360
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 11:18:42 2026 +0300

    optimize sampling offload (load VAE once, skip redundant transformer round-trip)

commit d2e8169
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 11:11:59 2026 +0300

    add orthogonal merging LoRA method

commit d967597
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 23 18:25:15 2026 +0300

    fix audio-only training (geometry-aware sigma sampling with cached virtual seq-len)

commit e360185
Merge: e8db31d 75151d0
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 23 14:22:37 2026 +0300

    bring back full fine-tuning scripts

commit e8db31d
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 23 11:22:50 2026 +0300

    add W8A8 activation quantization

commit e2faa52
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 23 00:59:23 2026 +0300

    add IC-LoRA (video-to-video) reference conditioning

commit 35d4918
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Feb 22 19:31:42 2026 +0300

    add NF4 4-bit quantization with LoftQ initialization and experimental AWQ activation-aware calibration

commit 70d8696
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 23:42:27 2026 +0300

    add gated attention support

commit ecf1896
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 22:51:53 2026 +0300

    improve FP8 layer exclusions (audio layer protection)

commit c1cc269
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 22:29:51 2026 +0300

    fix temporal VAE tiling

commit 2c313e5
Merge: 64d642e f65c153
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 20:24:43 2026 +0300

    Merge branch 'ltx-2-dev' into ltx-2

commit f65c153
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 17:26:24 2026 +0300

    fix for native LTX-2 LoHA support

commit 5a8136f
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 16:58:54 2026 +0300

    revert default mode to 'v'

commit 36fcc80
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 20 21:41:34 2026 +0300

    remove GUI prototype

commit 232c492
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 20 21:14:53 2026 +0300

    implement audio timestep and silence balancing features, update docs

commit 6469321
Merge: 74bca0f 0b2d692
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 20 19:35:38 2026 +0300

    Merge branch 'main' of https://github.com/kohya-ss/musubi-tuner into ltx-2-dev

commit 74bca0f
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 20 01:02:43 2026 +0300

    implement optional inverse-frequency audio loss weighting

commit 24df44b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Feb 19 22:46:32 2026 +0300

    add cache-only training option

commit a65a207
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 17 21:33:04 2026 +0300

    change default 'ltx2_mode' to 'av', remove deprecated ltx2_cache_latents.py

commit 66c1b1b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 17 11:24:38 2026 +0300

    remove FFT scripts from main branch

commit 75151d0
Merge: 90657a7 050c4cd
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 17 10:50:40 2026 +0300

    Merge branch 'ltx-2-dev' into ltx-2-ft-motion-preservation

commit 050c4cd
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 17 01:57:19 2026 +0300

    enhance LyCORIS support, restore --use_pinned_memory_for_block_swap flag, fix caching bugs

commit 0b2d692
Author: Kohya S. <52813779+kohya-ss@users.noreply.github.com>
Date:   Sun Feb 15 14:01:55 2026 +0900

    feat: add LoHa and LoKr network modules with architecture detection (kohya-ss#900)

    * feat: add LoHa and LoKr network modules with architecture detection

    * format: ruff

    * docs: update LoHa/LoKr section with links to algorithms and guidelines

    * fix: update regex matching to use fullmatch for pattern checks in LoRANetwork

    * feat: enhance LoRA conversion to support LoHa and LoKr formats

    * format: ruff

    * feat: extend convert_lora.py to support LoHa and LoKr format conversions for ComfyUI

    * Update src/musubi_tuner/convert_lora.py

    Fix typo.

    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

    * feat: update LoHa and LoKr modules to improve scale handling and dimension management

    * dic: update README and ascknowledgments

    ---------

    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

commit 3b845bd
Author: Kohya S. <52813779+kohya-ss@users.noreply.github.com>
Date:   Sun Feb 15 13:50:11 2026 +0900

    feat: Add block_swap_optimizer_patch_params option for supporting some optimizers with block swapping (kohya-ss#899)

    * feat: add block_swap_optimizer_patch_params option for optimizer compatibility with block swapping

    * format: ruff

    * feat: add --block_swap_optimizer_patch_params option for Z-Image fine-tuning compatibility with certain optimizers

commit eb2057e
Author: Kohya S <52813779+kohya-ss@users.noreply.github.com>
Date:   Sun Feb 15 09:16:53 2026 +0900

    fix: change del to None assignment to make it compatible with torch.compile

commit 64d642e
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 14 14:25:55 2026 +0300

    add dedicated LoRA merger and usage docs

commit 90657a7
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 14 11:28:59 2026 +0300

    replace synthetic anchors for motion replay with dataset-teacher motion replay, add probability gating

commit 542f0f6
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 13 16:10:06 2026 +0300

    add --caption_dropout_rate flag

commit 051455e
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 13 13:29:33 2026 +0300

    skip no-op attention masks to enable Flash Attention on cross-attn (optional)

commit 90dd9a6
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 13 13:24:41 2026 +0300

    add experimental full-FT motion-drift controls (anchor rehearsal, attention distillation, geometry-aware LR targeting)

    - add full fine-tune motion-preservation pipeline for image-only training: synthetic anchor clip builder + base-output anchor cache, motion rehearsal loss (temporal/full) with interval and weighting controls
    - add optional anchor-based attention self-distillation: capture/cache teacher attention maps on selected attn1 blocks, KL/MSE attention preservation loss during anchor rehearsal
    - add block-targeted full-FT optimizer grouping: freeze by range/index, per-block LR scale rules, non-block LR scale
    - add attention-geometry protection for to_q/to_k/q_norm/k_norm: extra LR scaling or full freeze for geometry params

commit dc43904
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 23:58:59 2026 +0300

    add histogram logging for Automagic optimizer

commit 71a4c18
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 21:19:39 2026 +0300

    fix: bounds check on lr_descriptions in generate_step_logs

commit eeb98f0
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 19:18:02 2026 +0300

    change default FPS to 25

commit 971c05d
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 02:46:07 2026 +0300

    fix bugs in full-rank finetuning scripts

commit 558c789
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 00:38:07 2026 +0300

    fix: normalize stereo layout in cache pipeline

commit b632711
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 10 23:52:26 2026 +0300

    add image conditioning for sampling, update GUI

commit 98754c5
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 10 11:55:10 2026 +0300

    implement opt-in audio-aware sampling (quota/probability)

commit e7493ca
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 10 02:29:28 2026 +0300

    add CREPA (WIP, not tested), modify GUI (scaffolding)

commit 00e255b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 9 02:58:08 2026 +0300

    add GUI dashboard (WIP)

commit 4ab2f1a
Merge: 5b53c20 1d223c7
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Feb 8 16:49:38 2026 +0300

    Merge branch 'ltx-2-slider' into ltx-2-dev

commit 5b53c20
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Feb 8 16:49:22 2026 +0300

    add optional preservation/regularization techniques for LoRA training

    - `blank prompt preservation` to regularize LoRA to preserve blank-prompt output (CFG baseline)
    - `differential output preservation` for class-prompt output
    - `prior divergence` to encourage LoRA to diverge from base model

commit 4fc3811
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 7 22:33:47 2026 +0300

    add information about validation datasets to docs, add Automagic optimizer
siraxe pushed a commit to siraxe/musubi-tuner that referenced this pull request Mar 15, 2026
@claude
Squashed commit of the following:
648d3dd 
commit 4d6e8f0
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 13 15:57:29 2026 +0300

    add Self-Flow temporal consistency losses, fix scheduling bugs, update GUI and docs

commit 5bd26d8
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 13 13:37:39 2026 +0300

    add optional per-dataset audio/video loss weight overrides

commit db3e2fe
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 13 13:23:43 2026 +0300

    update README

commit 18983cf
Merge: 25237ab ebaf126
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 13 13:00:49 2026 +0300

    Merge branch 'ltx-2-dev' into ltx-2

commit ebaf126
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 13 12:58:13 2026 +0300

    add web-based GUI dashboard for training configuration and monitoring (WIP)

commit 25237ab
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 12 09:51:39 2026 +0300

    fix V2V forward path for wrapped transformers, add --quantize_device CLI flag

    (cherry picked from commit 5f857f7)

commit dab632a
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 12 10:27:37 2026 +0300

    move Prompt-AdaLN modulation to fp32

commit 5f857f7
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 12 09:51:39 2026 +0300

    fix V2V forward path for wrapped transformers, add --quantize_device CLI flag

commit 409bd0e
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 12 00:24:29 2026 +0300

    fix inaccuracies in docs

commit f3621a6
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 11 00:08:48 2026 +0300

    add explicit Self-Flow layer ratio and projector LR knobs

commit 443039f
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 10 10:15:16 2026 +0300

    make exclude_patterns work without include_patterns

commit b0e36e3
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 10 09:58:03 2026 +0300

    fix caption dropout trainer-state bug (AkaneTendo25#44), guard FlashAttention masks

commit dd85eab
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 10 01:14:02 2026 +0300

    improve mixed-modality context validation

commit 2ca2745
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 10 00:45:53 2026 +0300

    fix mixed-modality conditioning, add explicit cache/checkpoint mismatch errors

commit 2baad17
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 9 23:18:34 2026 +0300

    update Self-Flow paper link

commit 1651801
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 9 23:06:24 2026 +0300

    fix AV no-audio fallback crash, harden context reshape (AkaneTendo25#43)

commit d3c3901
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 9 23:04:19 2026 +0300

    full fine-tuning: expand motion-preservation controls, fix attention replay behavior

commit 31c41cf
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 9 14:15:39 2026 +0300

    add full-finetune Self-Flow integration, add CPU offload support for Self-Flow teacher params

commit 9bd249d
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 9 09:43:36 2026 +0300

    extract LyCORIS runtime/config logic, add LyCORIS attach diagnostics

commit 1645425
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 9 01:34:54 2026 +0300

    update docs

commit 6245727
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 8 20:58:37 2026 +0300

    add NF4 pre-quantization tool with LoftQ support and auto-detection at load time (AkaneTendo25#42)

commit d7dc1d5
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 8 12:34:11 2026 +0300

    improve sampling-time LoRA/LyCORIS adapter detection and norm logging

commit 00bd605
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 8 00:22:19 2026 +0300

    add --loss_type argument (mae/huber/smooth_l1) as opt-in alternatives to MSE

commit b52b7aa
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 7 23:58:43 2026 +0300

    improve Self-Flow pipeline (non-breaking integration, EMA state resume, docs, and train script args)

commit 7011086
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 7 22:36:55 2026 +0300

    rework LyCORIS training (LTX2 preset targeting, FP8 compatibility, optimizer param handling)

commit 8eb3ff5
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 7 21:10:35 2026 +0300

    make fused backward accumulation-safe and Adafactor-only

commit a37aa4b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 7 00:35:35 2026 +0300

    initialize CREPA params before scheduler to avoid param-group mismatch

commit 997618f
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 6 21:12:06 2026 +0300

    stabilize audio forward path and document 2.3 fp8/cache/flash-attn troubleshooting

commit 30259ec
Merge: 1e0a49a 3d7bf14
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 6 13:20:54 2026 +0300

    Merge branch 'ltx-23' into ltx-2-dev

commit 1e0a49a
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 6 01:07:36 2026 +0300

    fix w8a8 bug

commit 3d7bf14
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 21:21:12 2026 +0300

    fix LTX-2.3 VAE decoder mismatch error (AkaneTendo25#40)

commit eae8c12
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 21:01:11 2026 +0300

    fix: materialize missing VAE buffers for LTX-2.3 compatibility

commit b6fe330
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 20:20:26 2026 +0300

    remove LTX-2 to 2.3 LoRA converter (AkaneTendo25#40)

commit 3c65e8c
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 18:58:26 2026 +0300

    implement Self-Flow training regularization

commit 9ded007
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 19:25:16 2026 +0300

    update docs

commit 7140a71
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 16:11:12 2026 +0300

    add LTX-2.3 metadata keys to full fine-tune and LoRA checkpoint metadata

commit 5be6108
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 15:38:00 2026 +0300

    add LTX-2.3 training infrastructure: version detection, stretched sigma sampler, CLI args, update docs (AkaneTendo25#40)

commit db6d575
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 13:47:00 2026 +0300

    fix: align transformer parity (prompt_adaln routing + cross-modal perturbation skip semantics)

commit 271395d
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 13:18:19 2026 +0300

    (experimental) add LTX-2 -> LTX-2.3 LoRA converter with shape-aware mapping and SVD fallback (AkaneTendo25#40)

commit cf5f673
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 11:28:42 2026 +0300

    (WIP) add LTX-2.3 compatibility: dual text projection, split connectors, cross-attn AdaLN, BWE vocoder (AkaneTendo25#40)

commit 908c40a
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 09:59:32 2026 +0300

    small doc fix

commit 9164339
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 20:39:57 2026 +0300

    enforce include_patterns as strict whitelist

commit c94c173
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 20:39:50 2026 +0300

    add physical audio-only transformer training option

commit f55704e
Merge: 1296186 1a23c66
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 14:20:10 2026 +0300

    Merge branch 'ltx-2' into ltx-2-dev

commit 1a23c66
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 14:19:44 2026 +0300

    fix non-relevant info in docs

commit 1296186
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 00:44:59 2026 +0300

    add opt-in block checkpointing improvements (pinned slab pool, eviction scoring, multi-block prefetch)

commit c5e0ad2
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 3 22:39:51 2026 +0300

    add per-module learning rate groups for LoRA training (AkaneTendo25#1)

commit fd87446
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 3 15:32:38 2026 +0300

    fix: audio channel coercion when neither dimension matches channel count (AkaneTendo25#12)

commit fc9fdbb
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 2 21:30:52 2026 +0300

    fix misleading docs

commit 36d4b35
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 2 13:52:40 2026 +0300

    stabilize audio-only sequence length and auto-detect audio mode

commit 3477108
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 2 11:16:12 2026 +0300

    update generate script

commit 794c9b6
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 22:03:45 2026 +0300

    update docs

commit 02db38a
Merge: 57bef9b 29318b1
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 20:41:48 2026 +0300

    Merge branch 'ltx-2' into ltx-2-dev

    # Conflicts:
    #	src/musubi_tuner/ltx2_train_network.py

commit 29318b1
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 20:38:23 2026 +0300

    add audio_bucket_strategy flag and fix audio-only mode issues (AkaneTendo25#12)

commit 96ae865
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 13:19:11 2026 +0300

    fix CREPA resume crash AkaneTendo25#37

commit 57bef9b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 13:19:11 2026 +0300

    fix CREPA resume crash AkaneTendo25#37

commit ef486aa
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 27 22:47:20 2026 +0300

    make subprocess audio decode default

commit 11526a7
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 27 20:28:07 2026 +0300

    add audio DOP, rename comfy output checkpoint to .comfy.safetensors, update docs

commit 9c373fe
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 27 12:10:15 2026 +0300

    fix audio_loss_balance_min default

commit 7fb3c61
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 27 11:54:55 2026 +0300

    skip FPS resampling for near-matching frame rates, save both original and Comfy LoRA by default

commit e0e1d13
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Feb 26 01:47:55 2026 +0300

    introduce experimental motion-prior replay and EWC regularization pipeline (FFT only)

commit 8cd3394
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 25 16:56:51 2026 +0300

    add --save_checkpoint_metadata for JSON sidecar files alongside checkpoints (AkaneTendo25#32)

commit 6bdd8aa
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 25 16:28:15 2026 +0300

    allow loading Gemma from safetensors (AkaneTendo25#20), change checkpoint cleanup behavior, rework standalone inference script

commit 672c487
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 15:55:33 2026 +0300

    make sample preview failures during FFT non-fatal

commit ecc2baa
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 14:50:26 2026 +0300

    update docs, remove redundant logging

commit 12f1121
Merge: 6205c57 d967597
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 12:13:00 2026 +0300

    Merge commit 'd967597f72aa70b507b97ed7c2aec7e9f315b87e' into ltx-2-dev

    # Conflicts:
    #	docs/ltx_2.md

commit 6205c57
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 11:23:30 2026 +0300

    add ComfyUI-compatible and merged checkpoint saving for full fine-tune, fix sampling and metadata errors

commit dc30360
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 11:18:42 2026 +0300

    optimize sampling offload (load VAE once, skip redundant transformer round-trip)

commit d2e8169
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 11:11:59 2026 +0300

    add orthogonal merging LoRA method

commit d967597
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 23 18:25:15 2026 +0300

    fix audio-only training (geometry-aware sigma sampling with cached virtual seq-len)

commit e360185
Merge: e8db31d 75151d0
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 23 14:22:37 2026 +0300

    bring back full fine-tuning scripts

commit e8db31d
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 23 11:22:50 2026 +0300

    add W8A8 activation quantization

commit e2faa52
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 23 00:59:23 2026 +0300

    add IC-LoRA (video-to-video) reference conditioning

commit 35d4918
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Feb 22 19:31:42 2026 +0300

    add NF4 4-bit quantization with LoftQ initialization and experimental AWQ activation-aware calibration

commit 70d8696
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 23:42:27 2026 +0300

    add gated attention support

commit ecf1896
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 22:51:53 2026 +0300

    improve FP8 layer exclusions (audio layer protection)

commit c1cc269
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 22:29:51 2026 +0300

    fix temporal VAE tiling

commit 2c313e5
Merge: 64d642e f65c153
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 20:24:43 2026 +0300

    Merge branch 'ltx-2-dev' into ltx-2

commit f65c153
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 17:26:24 2026 +0300

    fix for native LTX-2 LoHA support

commit 5a8136f
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 16:58:54 2026 +0300

    revert default mode to 'v'

commit 36fcc80
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 20 21:41:34 2026 +0300

    remove GUI prototype

commit 232c492
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 20 21:14:53 2026 +0300

    implement audio timestep and silence balancing features, update docs

commit 6469321
Merge: 74bca0f 0b2d692
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 20 19:35:38 2026 +0300

    Merge branch 'main' of https://github.com/kohya-ss/musubi-tuner into ltx-2-dev

commit 74bca0f
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 20 01:02:43 2026 +0300

    implement optional inverse-frequency audio loss weighting

commit 24df44b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Feb 19 22:46:32 2026 +0300

    add cache-only training option

commit a65a207
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 17 21:33:04 2026 +0300

    change default 'ltx2_mode' to 'av', remove deprecated ltx2_cache_latents.py

commit 66c1b1b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 17 11:24:38 2026 +0300

    remove FFT scripts from main branch

commit 75151d0
Merge: 90657a7 050c4cd
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 17 10:50:40 2026 +0300

    Merge branch 'ltx-2-dev' into ltx-2-ft-motion-preservation

commit 050c4cd
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 17 01:57:19 2026 +0300

    enhance LyCORIS support, restore --use_pinned_memory_for_block_swap flag, fix caching bugs

commit 0b2d692
Author: Kohya S. <52813779+kohya-ss@users.noreply.github.com>
Date:   Sun Feb 15 14:01:55 2026 +0900

    feat: add LoHa and LoKr network modules with architecture detection (kohya-ss#900)

    * feat: add LoHa and LoKr network modules with architecture detection

    * format: ruff

    * docs: update LoHa/LoKr section with links to algorithms and guidelines

    * fix: update regex matching to use fullmatch for pattern checks in LoRANetwork

    * feat: enhance LoRA conversion to support LoHa and LoKr formats

    * format: ruff

    * feat: extend convert_lora.py to support LoHa and LoKr format conversions for ComfyUI

    * Update src/musubi_tuner/convert_lora.py

    Fix typo.

    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

    * feat: update LoHa and LoKr modules to improve scale handling and dimension management

    * dic: update README and ascknowledgments

    ---------

    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

commit 3b845bd
Author: Kohya S. <52813779+kohya-ss@users.noreply.github.com>
Date:   Sun Feb 15 13:50:11 2026 +0900

    feat: Add block_swap_optimizer_patch_params option for supporting some optimizers with block swapping (kohya-ss#899)

    * feat: add block_swap_optimizer_patch_params option for optimizer compatibility with block swapping

    * format: ruff

    * feat: add --block_swap_optimizer_patch_params option for Z-Image fine-tuning compatibility with certain optimizers

commit eb2057e
Author: Kohya S <52813779+kohya-ss@users.noreply.github.com>
Date:   Sun Feb 15 09:16:53 2026 +0900

    fix: change del to None assignment to make it compatible with torch.compile

commit 64d642e
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 14 14:25:55 2026 +0300

    add dedicated LoRA merger and usage docs

commit 90657a7
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 14 11:28:59 2026 +0300

    replace synthetic anchors for motion replay with dataset-teacher motion replay, add probability gating

commit 542f0f6
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 13 16:10:06 2026 +0300

    add --caption_dropout_rate flag

commit 051455e
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 13 13:29:33 2026 +0300

    skip no-op attention masks to enable Flash Attention on cross-attn (optional)

commit 90dd9a6
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 13 13:24:41 2026 +0300

    add experimental full-FT motion-drift controls (anchor rehearsal, attention distillation, geometry-aware LR targeting)

    - add full fine-tune motion-preservation pipeline for image-only training: synthetic anchor clip builder + base-output anchor cache, motion rehearsal loss (temporal/full) with interval and weighting controls
    - add optional anchor-based attention self-distillation: capture/cache teacher attention maps on selected attn1 blocks, KL/MSE attention preservation loss during anchor rehearsal
    - add block-targeted full-FT optimizer grouping: freeze by range/index, per-block LR scale rules, non-block LR scale
    - add attention-geometry protection for to_q/to_k/q_norm/k_norm: extra LR scaling or full freeze for geometry params

commit dc43904
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 23:58:59 2026 +0300

    add histogram logging for Automagic optimizer

commit 71a4c18
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 21:19:39 2026 +0300

    fix: bounds check on lr_descriptions in generate_step_logs

commit eeb98f0
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 19:18:02 2026 +0300

    change default FPS to 25

commit 971c05d
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 02:46:07 2026 +0300

    fix bugs in full-rank finetuning scripts

commit 558c789
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 00:38:07 2026 +0300

    fix: normalize stereo layout in cache pipeline

commit b632711
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 10 23:52:26 2026 +0300

    add image conditioning for sampling, update GUI

commit 98754c5
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 10 11:55:10 2026 +0300

    implement opt-in audio-aware sampling (quota/probability)

commit e7493ca
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 10 02:29:28 2026 +0300

    add CREPA (WIP, not tested), modify GUI (scaffolding)

commit 00e255b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 9 02:58:08 2026 +0300

    add GUI dashboard (WIP)

commit 4ab2f1a
Merge: 5b53c20 1d223c7
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Feb 8 16:49:38 2026 +0300

    Merge branch 'ltx-2-slider' into ltx-2-dev

commit 5b53c20
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Feb 8 16:49:22 2026 +0300

    add optional preservation/regularization techniques for LoRA training

    - `blank prompt preservation` to regularize LoRA to preserve blank-prompt output (CFG baseline)
    - `differential output preservation` for class-prompt output
    - `prior divergence` to encourage LoRA to diverge from base model

commit 4fc3811
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 7 22:33:47 2026 +0300

    add information about validation datasets to docs, add Automagic optimizer
siraxe pushed a commit to siraxe/musubi-tuner that referenced this pull request Apr 18, 2026
@claude
Squashed commit of the following:
f1a19cc 
commit 7e05e92
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Apr 4 18:28:43 2026 +0300

    add ltx2_merge_lora_to_model.py, fix Relay LoRA docs

commit ec061a6
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Apr 4 12:19:44 2026 +0300

    fix attention enum crash on reassignment, fix LoRA weight converter fallback

commit cbe6217
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Apr 4 12:04:13 2026 +0300

    fix audio-only training crash on LTX-2.3

commit dd6ea16
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Apr 4 11:00:39 2026 +0300

    add TensorBoard timestep histograms

commit 9adf33f
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Apr 3 19:55:14 2026 +0300

    refactor ltx2_train_network.py

commit 6dd0ae4
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Apr 3 15:36:46 2026 +0300

    add av_ic combined video+audio IC-LoRA strategy, update docs

commit a1859ff
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Apr 3 12:23:16 2026 +0300

    add self-contained Windows installer script

commit c7fa83d
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Apr 3 09:36:31 2026 +0300

    make fp8/attention paths compile-safe, align audio decode dtype, fix connector training error

commit 5e76eba
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Apr 1 12:03:51 2026 +0300

    support warm-starting training from ComfyUI LoRAs (AkaneTendo25#55)

commit a151029
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Apr 1 11:37:12 2026 +0300

    fix AkaneTendo25#54

commit 8d46d77
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 29 19:50:26 2026 +0300

    support pre-quantized FP8 checkpoints, add dataset dir existence validation

commit 06153dd
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 29 17:04:29 2026 +0300

    fix grad norm tracking (accumulate on GPU to avoid per-param CUDA syncs)

commit 6da279f
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 29 16:33:18 2026 +0300

    fix docs

commit 1752963
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 29 15:07:44 2026 +0300

    add video_sa, video_sa_ff, video_sa_ca_ff LoRA target presets

commit 2834778
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 27 17:01:13 2026 +0300

    add audio/video dropout and cross-modal rank overrides

commit 76d287d
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 27 15:51:44 2026 +0300

    move HFATO degradation before noise addition for Self-Flow compatibility, implement OGM-GE loss balancing, per-group LR warmup overrides

commit ee9df03
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 26 23:42:49 2026 +0300

    fix mistake in docs

commit 8272caf
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 26 21:09:23 2026 +0300

    add audio slider training

commit a3e18d6
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 26 20:49:10 2026 +0300

    add first-frame conditioning for reference video sliders

commit 70977e2
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 26 17:21:20 2026 +0300

    add HFATO training objective

commit d470503
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 26 14:27:53 2026 +0300

    add LoRA support for text connector modules

commit fd60991
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 26 02:44:47 2026 +0300

    fix Self-Flow (schedule scope, LoHa/LoKr support, stale audio features, dual_timestep warning)

commit 72e6a15
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 26 02:29:40 2026 +0300

    add audio quality metrics module

commit b110989
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 24 15:33:10 2026 +0300

    add audio diagnostics (timestep-stratified loss, latent stats, SNR, loss ratio)

commit 5309031
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 24 12:27:26 2026 +0300

    fix identity guidance, add AV bimodal CFG for ID-LoRA sampling

commit 85fc1be
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 24 11:09:11 2026 +0300

    add TARP windowed cross-attention and DCR gradient routing for AV training

commit 3126a73
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 24 10:11:25 2026 +0300

    turn off ffn_clamp

commit dd96141
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 21 23:42:05 2026 +0300

    fix audio_ref_only_ic inference masks and position layout, add warnings

commit fad7272
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 21 17:41:02 2026 +0300

    add FFN output clamping for bf16 overflow prevention

commit e92f3dc
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 21 17:08:07 2026 +0300

    add Self-Flow EMA drift metric and configurable projector activation

commit c6ef494
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 21 16:34:46 2026 +0300

    add Cross-Task Synergy auxiliary losses for joint audio-video training

commit 5d4b8fa
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 21 15:55:37 2026 +0300

    add G2D modality freezing

commit 8236bd0
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 21 15:48:24 2026 +0300

    add independent modality dropout

commit adbb8ba
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 21 15:35:20 2026 +0300

    add Self-Flow audio alignment

commit 6b2f142
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 21 12:05:00 2026 +0300

    add uncertainty loss weighting for AV training

commit 21ed691
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 21 11:36:59 2026 +0300

    add per-modality gradient norm logging

commit bb699e0
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 21 11:25:16 2026 +0300

    add per-modality LoRA rank

commit cb514c3
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 21 02:09:41 2026 +0300

    enable Self-Flow in AV mode (video branch only), fix AV teacher-forward timestep handling

commit 5ddca9a
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 20 23:42:14 2026 +0300

    guard empty pinned swap jobs, drop deprecated method

commit 13471cb
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 19 20:14:30 2026 +0300

    add resume metadata, mid-epoch resume, optimizer reset flags, loss continuity

commit 81a8a47
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 19 03:28:03 2026 +0300

    remove Python 3.13 upper bound and unpin safetensors for 3.13 wheel support (AkaneTendo25#49)

commit 36991a1
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 18 21:08:59 2026 +0300

    update docs/ltx_2.md

commit 5af5ec5
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 18 15:51:05 2026 +0300

    improve dashboard VRAM estimator accuracy

commit b8207d4
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 16 23:47:07 2026 +0300

    add --shifted_logit_shift override parameter

commit 3569e01
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 16 23:34:35 2026 +0300

    add content-seeded motion priors and warmup scheduling for FFT pipeline

commit 30587ec
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 16 14:44:48 2026 +0300

    add troubleshooting entry for transformers 5.x Gemma rotary buffer mismatch

commit d3cbe05
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 15 23:15:50 2026 +0300

    add --autoresume flag for automatic training state recovery

commit ba16c69
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 15 14:38:04 2026 +0300

    fix Self-Flow FFT binding, add motion priors validation, improve docs

commit d0dab19
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 15 00:37:20 2026 +0300

    update Self-Flow implementation (base-model teacher, unified lambda scheduling, fix temporal feature space, mask-focused loss)

commit 3e8e366
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 14 21:17:57 2026 +0300

    add early shape validation guards for AV text embedding pipeline

commit be132bd
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 14 17:11:33 2026 +0300

    fix video encoder crash on mixed audio+video dataset caching, clean up docs example

commit fe17466
Merge: 25f9273 a850208
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 14 16:25:37 2026 +0300

    Merge branch 'ltx-2-dev' of https://github.com/AkaneTendo25/musubi-tuner into ltx-2-dev

commit 25f9273
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 14 15:40:39 2026 +0300

    add IC-LoRA GUI controls, expand docs community links

commit a850208
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 14 15:40:39 2026 +0300

    add IC-LoRA strategy selector and audio-reference IC-LoRA controls to GUI

commit fd381db
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 14 15:40:27 2026 +0300

    fix regressive crash on AV context split for LTX-2.0 checkpoints

commit 1236c02
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 14 11:53:31 2026 +0300

    add --loss_type support (MSE/MAE/Huber), expand docs references

commit 8146a41
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 14 11:03:19 2026 +0300

    add audio-only IC-LoRA training support

commit 4a434dc
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 14 10:06:22 2026 +0300

    add audio-reference IC-LoRA training and sampling support

commit 2a6acab
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 14 01:14:17 2026 +0300

    fix audio_loss guard mismatch

commit 3ff1c0a
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 13 19:46:59 2026 +0300

    fix dead caption_channels validation, add LTX-2.3 video mode check, clean up self-flow has_active_loss

commit 4d6e8f0
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 13 15:57:29 2026 +0300

    add Self-Flow temporal consistency losses, fix scheduling bugs, update GUI and docs

commit 5bd26d8
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 13 13:37:39 2026 +0300

    add optional per-dataset audio/video loss weight overrides

commit db3e2fe
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 13 13:23:43 2026 +0300

    update README

commit 18983cf
Merge: 25237ab ebaf126
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 13 13:00:49 2026 +0300

    Merge branch 'ltx-2-dev' into ltx-2

commit ebaf126
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 13 12:58:13 2026 +0300

    add web-based GUI dashboard for training configuration and monitoring (WIP)

commit 25237ab
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 12 09:51:39 2026 +0300

    fix V2V forward path for wrapped transformers, add --quantize_device CLI flag

    (cherry picked from commit 5f857f7)

commit dab632a
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 12 10:27:37 2026 +0300

    move Prompt-AdaLN modulation to fp32

commit 5f857f7
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 12 09:51:39 2026 +0300

    fix V2V forward path for wrapped transformers, add --quantize_device CLI flag

commit 409bd0e
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 12 00:24:29 2026 +0300

    fix inaccuracies in docs

commit f3621a6
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 11 00:08:48 2026 +0300

    add explicit Self-Flow layer ratio and projector LR knobs

commit 443039f
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 10 10:15:16 2026 +0300

    make exclude_patterns work without include_patterns

commit b0e36e3
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 10 09:58:03 2026 +0300

    fix caption dropout trainer-state bug (AkaneTendo25#44), guard FlashAttention masks

commit dd85eab
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 10 01:14:02 2026 +0300

    improve mixed-modality context validation

commit 2ca2745
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 10 00:45:53 2026 +0300

    fix mixed-modality conditioning, add explicit cache/checkpoint mismatch errors

commit 2baad17
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 9 23:18:34 2026 +0300

    update Self-Flow paper link

commit 1651801
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 9 23:06:24 2026 +0300

    fix AV no-audio fallback crash, harden context reshape (AkaneTendo25#43)

commit d3c3901
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 9 23:04:19 2026 +0300

    full fine-tuning: expand motion-preservation controls, fix attention replay behavior

commit 31c41cf
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 9 14:15:39 2026 +0300

    add full-finetune Self-Flow integration, add CPU offload support for Self-Flow teacher params

commit 9bd249d
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 9 09:43:36 2026 +0300

    extract LyCORIS runtime/config logic, add LyCORIS attach diagnostics

commit 1645425
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 9 01:34:54 2026 +0300

    update docs

commit 6245727
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 8 20:58:37 2026 +0300

    add NF4 pre-quantization tool with LoftQ support and auto-detection at load time (AkaneTendo25#42)

commit d7dc1d5
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 8 12:34:11 2026 +0300

    improve sampling-time LoRA/LyCORIS adapter detection and norm logging

commit 00bd605
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 8 00:22:19 2026 +0300

    add --loss_type argument (mae/huber/smooth_l1) as opt-in alternatives to MSE

commit b52b7aa
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 7 23:58:43 2026 +0300

    improve Self-Flow pipeline (non-breaking integration, EMA state resume, docs, and train script args)

commit 7011086
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 7 22:36:55 2026 +0300

    rework LyCORIS training (LTX2 preset targeting, FP8 compatibility, optimizer param handling)

commit 8eb3ff5
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 7 21:10:35 2026 +0300

    make fused backward accumulation-safe and Adafactor-only

commit a37aa4b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Mar 7 00:35:35 2026 +0300

    initialize CREPA params before scheduler to avoid param-group mismatch

commit 997618f
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 6 21:12:06 2026 +0300

    stabilize audio forward path and document 2.3 fp8/cache/flash-attn troubleshooting

commit 30259ec
Merge: 1e0a49a 3d7bf14
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 6 13:20:54 2026 +0300

    Merge branch 'ltx-23' into ltx-2-dev

commit 1e0a49a
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Mar 6 01:07:36 2026 +0300

    fix w8a8 bug

commit 3d7bf14
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 21:21:12 2026 +0300

    fix LTX-2.3 VAE decoder mismatch error (AkaneTendo25#40)

commit eae8c12
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 21:01:11 2026 +0300

    fix: materialize missing VAE buffers for LTX-2.3 compatibility

commit b6fe330
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 20:20:26 2026 +0300

    remove LTX-2 to 2.3 LoRA converter (AkaneTendo25#40)

commit 3c65e8c
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 18:58:26 2026 +0300

    implement Self-Flow training regularization

commit 9ded007
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 19:25:16 2026 +0300

    update docs

commit 7140a71
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 16:11:12 2026 +0300

    add LTX-2.3 metadata keys to full fine-tune and LoRA checkpoint metadata

commit 5be6108
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 15:38:00 2026 +0300

    add LTX-2.3 training infrastructure: version detection, stretched sigma sampler, CLI args, update docs (AkaneTendo25#40)

commit db6d575
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 13:47:00 2026 +0300

    fix: align transformer parity (prompt_adaln routing + cross-modal perturbation skip semantics)

commit 271395d
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 13:18:19 2026 +0300

    (experimental) add LTX-2 -> LTX-2.3 LoRA converter with shape-aware mapping and SVD fallback (AkaneTendo25#40)

commit cf5f673
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 11:28:42 2026 +0300

    (WIP) add LTX-2.3 compatibility: dual text projection, split connectors, cross-attn AdaLN, BWE vocoder (AkaneTendo25#40)

commit 908c40a
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Mar 5 09:59:32 2026 +0300

    small doc fix

commit 9164339
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 20:39:57 2026 +0300

    enforce include_patterns as strict whitelist

commit c94c173
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 20:39:50 2026 +0300

    add physical audio-only transformer training option

commit f55704e
Merge: 1296186 1a23c66
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 14:20:10 2026 +0300

    Merge branch 'ltx-2' into ltx-2-dev

commit 1a23c66
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 14:19:44 2026 +0300

    fix non-relevant info in docs

commit 1296186
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Mar 4 00:44:59 2026 +0300

    add opt-in block checkpointing improvements (pinned slab pool, eviction scoring, multi-block prefetch)

commit c5e0ad2
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 3 22:39:51 2026 +0300

    add per-module learning rate groups for LoRA training (AkaneTendo25#1)

commit fd87446
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Mar 3 15:32:38 2026 +0300

    fix: audio channel coercion when neither dimension matches channel count (AkaneTendo25#12)

commit fc9fdbb
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 2 21:30:52 2026 +0300

    fix misleading docs

commit 36d4b35
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 2 13:52:40 2026 +0300

    stabilize audio-only sequence length and auto-detect audio mode

commit 3477108
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Mar 2 11:16:12 2026 +0300

    update generate script

commit 794c9b6
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 22:03:45 2026 +0300

    update docs

commit 02db38a
Merge: 57bef9b 29318b1
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 20:41:48 2026 +0300

    Merge branch 'ltx-2' into ltx-2-dev

    # Conflicts:
    #	src/musubi_tuner/ltx2_train_network.py

commit 29318b1
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 20:38:23 2026 +0300

    add audio_bucket_strategy flag and fix audio-only mode issues (AkaneTendo25#12)

commit 96ae865
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 13:19:11 2026 +0300

    fix CREPA resume crash AkaneTendo25#37

commit 57bef9b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Mar 1 13:19:11 2026 +0300

    fix CREPA resume crash AkaneTendo25#37

commit ef486aa
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 27 22:47:20 2026 +0300

    make subprocess audio decode default

commit 11526a7
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 27 20:28:07 2026 +0300

    add audio DOP, rename comfy output checkpoint to .comfy.safetensors, update docs

commit 9c373fe
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 27 12:10:15 2026 +0300

    fix audio_loss_balance_min default

commit 7fb3c61
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 27 11:54:55 2026 +0300

    skip FPS resampling for near-matching frame rates, save both original and Comfy LoRA by default

commit e0e1d13
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Feb 26 01:47:55 2026 +0300

    introduce experimental motion-prior replay and EWC regularization pipeline (FFT only)

commit 8cd3394
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 25 16:56:51 2026 +0300

    add --save_checkpoint_metadata for JSON sidecar files alongside checkpoints (AkaneTendo25#32)

commit 6bdd8aa
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 25 16:28:15 2026 +0300

    allow loading Gemma from safetensors (AkaneTendo25#20), change checkpoint cleanup behavior, rework standalone inference script

commit 672c487
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 15:55:33 2026 +0300

    make sample preview failures during FFT non-fatal

commit ecc2baa
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 14:50:26 2026 +0300

    update docs, remove redundant logging

commit 12f1121
Merge: 6205c57 d967597
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 12:13:00 2026 +0300

    Merge commit 'd967597f72aa70b507b97ed7c2aec7e9f315b87e' into ltx-2-dev

    # Conflicts:
    #	docs/ltx_2.md

commit 6205c57
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 11:23:30 2026 +0300

    add ComfyUI-compatible and merged checkpoint saving for full fine-tune, fix sampling and metadata errors

commit dc30360
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 11:18:42 2026 +0300

    optimize sampling offload (load VAE once, skip redundant transformer round-trip)

commit d2e8169
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 24 11:11:59 2026 +0300

    add orthogonal merging LoRA method

commit d967597
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 23 18:25:15 2026 +0300

    fix audio-only training (geometry-aware sigma sampling with cached virtual seq-len)

commit e360185
Merge: e8db31d 75151d0
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 23 14:22:37 2026 +0300

    bring back full fine-tuning scripts

commit e8db31d
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 23 11:22:50 2026 +0300

    add W8A8 activation quantization

commit e2faa52
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 23 00:59:23 2026 +0300

    add IC-LoRA (video-to-video) reference conditioning

commit 35d4918
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Feb 22 19:31:42 2026 +0300

    add NF4 4-bit quantization with LoftQ initialization and experimental AWQ activation-aware calibration

commit 70d8696
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 23:42:27 2026 +0300

    add gated attention support

commit ecf1896
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 22:51:53 2026 +0300

    improve FP8 layer exclusions (audio layer protection)

commit c1cc269
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 22:29:51 2026 +0300

    fix temporal VAE tiling

commit 2c313e5
Merge: 64d642e f65c153
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 20:24:43 2026 +0300

    Merge branch 'ltx-2-dev' into ltx-2

commit f65c153
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 17:26:24 2026 +0300

    fix for native LTX-2 LoHA support

commit 5a8136f
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 21 16:58:54 2026 +0300

    revert default mode to 'v'

commit 36fcc80
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 20 21:41:34 2026 +0300

    remove GUI prototype

commit 232c492
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 20 21:14:53 2026 +0300

    implement audio timestep and silence balancing features, update docs

commit 6469321
Merge: 74bca0f 0b2d692
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 20 19:35:38 2026 +0300

    Merge branch 'main' of https://github.com/kohya-ss/musubi-tuner into ltx-2-dev

commit 74bca0f
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 20 01:02:43 2026 +0300

    implement optional inverse-frequency audio loss weighting

commit 24df44b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Thu Feb 19 22:46:32 2026 +0300

    add cache-only training option

commit a65a207
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 17 21:33:04 2026 +0300

    change default 'ltx2_mode' to 'av', remove deprecated ltx2_cache_latents.py

commit 66c1b1b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 17 11:24:38 2026 +0300

    remove FFT scripts from main branch

commit 75151d0
Merge: 90657a7 050c4cd
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 17 10:50:40 2026 +0300

    Merge branch 'ltx-2-dev' into ltx-2-ft-motion-preservation

commit 050c4cd
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 17 01:57:19 2026 +0300

    enhance LyCORIS support, restore --use_pinned_memory_for_block_swap flag, fix caching bugs

commit 0b2d692
Author: Kohya S. <52813779+kohya-ss@users.noreply.github.com>
Date:   Sun Feb 15 14:01:55 2026 +0900

    feat: add LoHa and LoKr network modules with architecture detection (kohya-ss#900)

    * feat: add LoHa and LoKr network modules with architecture detection

    * format: ruff

    * docs: update LoHa/LoKr section with links to algorithms and guidelines

    * fix: update regex matching to use fullmatch for pattern checks in LoRANetwork

    * feat: enhance LoRA conversion to support LoHa and LoKr formats

    * format: ruff

    * feat: extend convert_lora.py to support LoHa and LoKr format conversions for ComfyUI

    * Update src/musubi_tuner/convert_lora.py

    Fix typo.

    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

    * feat: update LoHa and LoKr modules to improve scale handling and dimension management

    * dic: update README and ascknowledgments

    ---------

    Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

commit 3b845bd
Author: Kohya S. <52813779+kohya-ss@users.noreply.github.com>
Date:   Sun Feb 15 13:50:11 2026 +0900

    feat: Add block_swap_optimizer_patch_params option for supporting some optimizers with block swapping (kohya-ss#899)

    * feat: add block_swap_optimizer_patch_params option for optimizer compatibility with block swapping

    * format: ruff

    * feat: add --block_swap_optimizer_patch_params option for Z-Image fine-tuning compatibility with certain optimizers

commit eb2057e
Author: Kohya S <52813779+kohya-ss@users.noreply.github.com>
Date:   Sun Feb 15 09:16:53 2026 +0900

    fix: change del to None assignment to make it compatible with torch.compile

commit 64d642e
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 14 14:25:55 2026 +0300

    add dedicated LoRA merger and usage docs

commit 90657a7
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 14 11:28:59 2026 +0300

    replace synthetic anchors for motion replay with dataset-teacher motion replay, add probability gating

commit 542f0f6
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 13 16:10:06 2026 +0300

    add --caption_dropout_rate flag

commit 051455e
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 13 13:29:33 2026 +0300

    skip no-op attention masks to enable Flash Attention on cross-attn (optional)

commit 90dd9a6
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Fri Feb 13 13:24:41 2026 +0300

    add experimental full-FT motion-drift controls (anchor rehearsal, attention distillation, geometry-aware LR targeting)

    - add full fine-tune motion-preservation pipeline for image-only training: synthetic anchor clip builder + base-output anchor cache, motion rehearsal loss (temporal/full) with interval and weighting controls
    - add optional anchor-based attention self-distillation: capture/cache teacher attention maps on selected attn1 blocks, KL/MSE attention preservation loss during anchor rehearsal
    - add block-targeted full-FT optimizer grouping: freeze by range/index, per-block LR scale rules, non-block LR scale
    - add attention-geometry protection for to_q/to_k/q_norm/k_norm: extra LR scaling or full freeze for geometry params

commit dc43904
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 23:58:59 2026 +0300

    add histogram logging for Automagic optimizer

commit 71a4c18
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 21:19:39 2026 +0300

    fix: bounds check on lr_descriptions in generate_step_logs

commit eeb98f0
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 19:18:02 2026 +0300

    change default FPS to 25

commit 971c05d
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 02:46:07 2026 +0300

    fix bugs in full-rank finetuning scripts

commit 558c789
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Wed Feb 11 00:38:07 2026 +0300

    fix: normalize stereo layout in cache pipeline

commit b632711
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 10 23:52:26 2026 +0300

    add image conditioning for sampling, update GUI

commit 98754c5
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 10 11:55:10 2026 +0300

    implement opt-in audio-aware sampling (quota/probability)

commit e7493ca
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Tue Feb 10 02:29:28 2026 +0300

    add CREPA (WIP, not tested), modify GUI (scaffolding)

commit 00e255b
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Mon Feb 9 02:58:08 2026 +0300

    add GUI dashboard (WIP)

commit 4ab2f1a
Merge: 5b53c20 1d223c7
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Feb 8 16:49:38 2026 +0300

    Merge branch 'ltx-2-slider' into ltx-2-dev

commit 5b53c20
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sun Feb 8 16:49:22 2026 +0300

    add optional preservation/regularization techniques for LoRA training

    - `blank prompt preservation` to regularize LoRA to preserve blank-prompt output (CFG baseline)
    - `differential output preservation` for class-prompt output
    - `prior divergence` to encourage LoRA to diverge from base model

commit 4fc3811
Author: AkaneTendo25 <seruva19@mail.ru>
Date:   Sat Feb 7 22:33:47 2026 +0300

    add information about validation datasets to docs, add Automagic optimizer
DustyL added a commit to DustyL/blissful-tuner that referenced this pull request May 24, 2026
@DustyL @kohya-ss
@christopher5106 @obsxrver @belambert @claude @rockerBOO
chore: fold upstream musubi-tuner v0.3.0 onto blissful training surfa…
37a2e1c 
…ce (#8)

* feat: Qwen-Image-Layered support (kohya-ss#816)

* feat: Qwen-Image-Layered inference support

* fix: version no

* refactor: check and format by ruff because diffrence from upstream is large

* feat: add Qwen-Image-Layered support with necessary modifications (WIP)

* feat: update Qwen-Image-Layered training (WIP)

* feat: add multiple_target field to ConfigSanitizer and update dataset group generation

* feat: make sure non-layered architectures got content withtout alpha

* feat: add Qwen-Image-Layered support and update training documentation

* doc: update documentation for Qwen-Image-Layered training and configuration

* feat: update sample image alignment vertically to horizontally

* fix: interactive generation

* feat: `multiple_target` should be in `[[datasets]]` section

* fix: multiple-target dataset in ImageJsonlDatasource

* fix: RGB (not RGBA) image handling in caching latents

* fix: ensure 3-channel content in encode_datasets_framepack function

* Update src/musubi_tuner/qwen_image_train_network.py

remove unnecessary del statement

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/musubi_tuner/qwen_image_train_network.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/musubi_tuner/qwen_image_generate_image.py

Update comment.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/musubi_tuner/qwen_image_train_network.py

Add a latent shape validation check for layered in `call_dit`.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/musubi_tuner/qwen_image_cache_latents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/musubi_tuner/dataset/image_video_dataset.py

Clarify the error message.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/musubi_tuner/qwen_image/qwen_image_utils.py

Fix shape for randn tensor (reshaped after this though).

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/musubi_tuner/qwen_image_train_network.py

Change img_shapes to list from list of lists.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/musubi_tuner/qwen_image_generate_image.py

Fix assertion for non-edit/layered model.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: correct typo in variable name for rounded_target_frames, make sure target frames are not duplicated

* format: ruff

* feat: add RGB conversion for images in get_qwen_prompt_embeds_with_image and ensure 3-channel output for non-layered control images

* doc: update README and overview.md for Qwen-Image-Layered

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: load control image for qwen image layered in generation

* fix: update mu calculation in generate function to use control_latent shape

* fix: ensure img_shapes is consistently formatted as a list of lists for normal/layered images

* fix: update LoRA conversion script and documentation for Diffusers format compatibility

* doc: update Z-Image LoRA conversion to use convert_lora.py for improved compatibility with ComfyUI

* feat: Remove the restriction that the original image must be the first layer during training. (kohya-ss#852)

* feat: add option to remove first image from target in layered model preprocessing

* feat: move removing option from caching to training

* doc: add option to exclude first target image during Qwen-Image-Layered training

* doc: add an option to remove the first image from target for Qwen-Image-Layered

* doc: clarify output layer specification for Qwen-Image-Layered when removing first image from target

* fix: batch generation degradation for Qwen-Image

* Flux.2-dev training (kohya-ss#841)

* Flux 2 dev training

* feat(image-ml): wip

* feat(image-ml): wip

* feat(image-ml): wip

* feat(image-ml): wip

* feat(image-ml): wip

* feat(image-ml): wip

* feat(image-ml): wip

* feat(image-ml): wip

* adding Flux Klein

* feat(image-ml): wip

* feat(image-ml): wip

* feat(image-ml): wip

* feat(image-ml): wip

* feat(image-ml): wip

* Fix flux 2 formatting and update readme (kohya-ss#857)

* format: by ruff

* fix: README updates for FLUX.2 LoRA training support and documentation

* fix: by ruff etc.

* docs: Update README to credit contributor with full affiliation

* FLUX.2 Improvements and fixes (kohya-ss#858)

* fix: generation for klein

* fix: correct function name for loading text embedder and add model version to flow model loading

* feat: use fp8 for Qwen3 on generation

* feat: add guidance scale argument for classifier free guidance

* feat(image-ml): fix (kohya-ss#864)

* feat: enhance model loading with FP8 optimization and LoRA support

* fix: ruff

* feat: use ARB cropping instead of center cropping

* feat: improve num of block swap logic in Flux2 model

* fix: ruff check

* feat: change model info to class and use different architecture for dev/klein 4b/klein 9b

* feat: standardize control image resizing parameters and deprecate old keys (WIP)

* feat: update control image resizing options, and add FLUX.2 settings

* feat: remove resizing for control images from latent caching in Flux2

* feat: update Flux2 model handling and improve control image processing

* feat: add support for negative prompts in text input processing and update denoising functions

* feat: remove unused image processing functions to streamline codebase

* feat: refactor Flux2 model and use unified attntion

* feat: update model version identifiers for simplicity and correct doc

* feat: inherit Klein9BParams and Klein4BParams from Flux2Params for consistency

* feat: extend model version options in argument parser for enhanced flexibility

* fix: move qwen_variant declaration to last

* feat: enhance show_image and show_console functions to support control images

* fix: remove redundant content stacking in preprocess_contents_flux_2 function

* feat: update comments to reflect support for Qwen-3 text encoder in main function

* doc: add --model_version option for FLUX.2 scripts with default to dev

* fix: do not use batched_prc_img

* feat: remove alias for FLUX.2 model name

* fix: denoise_cfg doen't work, and add autocast for generation with bf16 models

* fix: improve memory management by forcing garbage collection and optimizing tensor device transfers

* fix: vae_dtype works for training

* feat: make default discrete flow shift for FLUX.2 model

* fix: remove debug print statement for use_guidance_embed in Flux2 class

* fix: update memory saving options and add vae_dtype specification for FLUX.2

* docs: update FLUX.2 model download instructions and add Qwen3 4B weight usage details

* feat: add vae_dtype option to work with cache latents

* fix: backward causes an error due to lack of casting

* doc: update model sampling parameters in documentation

* docs: add alternative options for memory optimization in FLUX.2 documentation

* refactor: clean up unused variables and streamline attention function

* docs: update FLUX.2 documentation with new memory optimization options and model swap block limits
fix: correct dtype for text encoder and improve prompt caching logic in training script

* fix: correct tensor device and dtype casting in control latent encoding

* Update src/musubi_tuner/flux_2_generate_image.py

fix dtype

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/musubi_tuner/flux_2_generate_image.py

Fix compile in generation script.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/musubi_tuner/flux_2_generate_image.py

Fix fp8 dtype.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/musubi_tuner/flux_2_generate_image.py

Return model from load_dit_model when merging.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/musubi_tuner/flux_2_generate_image.py

Use del instead of assigning None.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: autocast for fp8, batch generation

* fix: rename variable for clarity in control latent processing

* fix: remove unused parameter from generate function

* docs: add new architecture links to README files

* fix: update introduction to include additional architectures in README files

* docs: add model version specification and training recommendations to flux_2.md

* fix: update module name conversion for HunyuanVideo and FLUX lora formats

* fix: add note about insufficient testing for klein 9B and dev models in README

* fix: remove outdated notes on unimplemented features and untested models from README

---------

Co-authored-by: Christopher <christopher.bourez@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: move image tensor to GPU for gradient checkpointing CPU offloading

* feat: validate fp8 format in load_safetensors_with_fp8_optimization

* chore: add references directory to .gitignore to store temp reference files for AI agents.

* docs: update overview.md for clarity and remove redundant commands

* update link to caption_images_by_qwen_vl.py

Updated the link to use the absolute path to src/musubi_tuner/caption_images_by_qwen_vl.py instead of ./

* fix: Update CFG handling in sample generation for Z-Image (kohya-ss#878)

* fix: update guidance scale handling and model training instructions for Z-Image

* fix: update Z-Image tokenizer ID to base version

* doc: update for Z-Image-Base

* fix: unpack_latents function to support 1 layer latent for QIL

* fix: change del to None assignment to make it compatible with torch.compile

* feat: Add block_swap_optimizer_patch_params option for supporting some optimizers with block swapping (kohya-ss#899)

* feat: add block_swap_optimizer_patch_params option for optimizer compatibility with block swapping

* format: ruff

* feat: add --block_swap_optimizer_patch_params option for Z-Image fine-tuning compatibility with certain optimizers

* feat: add LoHa and LoKr network modules with architecture detection (kohya-ss#900)

* feat: add LoHa and LoKr network modules with architecture detection

* format: ruff

* docs: update LoHa/LoKr section with links to algorithms and guidelines

* fix: update regex matching to use fullmatch for pattern checks in LoRANetwork

* feat: enhance LoRA conversion to support LoHa and LoKr formats

* format: ruff

* feat: extend convert_lora.py to support LoHa and LoKr format conversions for ComfyUI

* Update src/musubi_tuner/convert_lora.py

Fix typo.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* feat: update LoHa and LoKr modules to improve scale handling and dimension management

* dic: update README and ascknowledgments

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(qwen_image_cache_latents): print -> log.debug in decode debug block

* Refactor/trainer split (kohya-ss#930)

* refactor(training): extract shared helpers from hv_train_network into training package

Move free functions out of hv_train_network.py into a new musubi_tuner.training
subpackage so that each file matches its contents by name:

- training/accelerator_setup.py: clean_memory_on_device, collator_class, prepare_accelerator
- training/sampling_prompts.py:  line_to_prompt_dict, load_prompts, should_sample_images
- training/timesteps.py:         compute_density_for_timestep_sampling, get_sigmas, compute_loss_weighting_for_sd3
- training/parser_common.py:     setup_parser_common, read_config_from_file

hv_train_network.py re-exports these symbols so that existing imports from
wan_train_network.py and other architecture-specific scripts keep working
unchanged. NetworkTrainer itself is left in place for a follow-up commit.

hv_train_network.py: 3096 -> 2094 lines.

* refactor(training): move NetworkTrainer to training.trainer_base

Split the NetworkTrainer class out of hv_train_network.py into its own module
so the file name matches its contents. hv_train_network.py is now just the
HunyuanVideo-specific entry point (hv_setup_parser + main) plus re-exports
for backward compatibility with architecture-specific training scripts that
import NetworkTrainer / helpers from musubi_tuner.hv_train_network.

SS_METADATA_* constants move with NetworkTrainer (it uses them) and are
re-exported from hv_train_network.py.

Fix sys.path fallback for --network_module short names: __file__ now lives
under musubi_tuner/training/, so step one extra level up to keep
`--network_module networks.lora` resolving to musubi_tuner/networks/lora.py.

hv_train_network.py: 2094 -> 87 lines.
training/trainer_base.py: new, 2061 lines.

* refactor(training): split NetworkTrainer.train() into phase methods

Decompose the 693-line train() into private per-phase methods so the top
of the method reads as a table of contents:

- _validate_args_and_init
- _init_session
- _build_dataset
- _prepare_accelerator_and_dtypes
- _prepare_sampling
- _load_dit_and_swap
- _build_network
- _build_optimizer_and_dataloader
- _prepare_with_accelerator
- _register_hooks_and_resume
- _run_training_loop

No behavior change: state flows through return tuples and arguments;
existing self attributes (blocks_to_swap, num_timestep_buckets) are kept.
Verified with --help and by importing all *_train_network.py subclasses;
loss curves over 20 steps on Z-Image match main/Step 1 within numerical
error.

* refactor(training): split setup_parser_common into category helpers

Decompose the 690-line setup_parser_common into _add_*_args helpers
grouped by CLI category (general, attention, compile/dynamo, training,
logging, ddp, sampling, optimizer, lr_scheduler, memory, timestep,
network, save_load, metadata, huggingface, model). setup_parser_common
now reads as a table of contents.

No behavioral change: the resulting parser's argument set is identical
(verified by sorted argument diff). Only --help display order differs —
dynamo args now sit next to compile args instead of after --fp8_base.
The nested int_or_float is lifted to module scope as _int_or_float.

* fix(training): add __all__ to hv_train_network and remove unused imports from trainer_base

hv_train_network re-exports symbols for backward compatibility with
architecture-specific scripts. Adding __all__ suppresses ruff F401
warnings while documenting the intentional re-exports. Also remove
11 unused imports from trainer_base left over from the refactoring.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* Refactor/dataset split (kohya-ss#931)

* refactor: split image_video_dataset.py into focused modules (2417 -> 901 lines)

Extract five modules from the monolithic dataset file for better readability:
- architectures.py: ARCHITECTURE_* constants
- media_utils.py: image/video file I/O utilities (glob, load, resize)
- cache_io.py: latent/text encoder cache save functions (all architectures)
- bucket.py: BucketSelector and BucketBatchManager
- datasources.py: Content/Image/Video Datasource class hierarchy

Backward compatibility maintained via re-exports in image_video_dataset.py.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: resolve ruff warnings in image_video_dataset.py

Remove unused `math` import and redundant explicit imports from
media_utils. Add explicit imports for ARCHITECTURE_* constants used
locally to satisfy F405 while keeping star imports for re-export.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* Refactor: move HunyuanVideo-specific defaults out of NetworkTrainer base

trainer_base.py had HunyuanVideo-specific implementations (load_vae,
load_transformer, call_dit, process_sample_prompts, do_inference, ...) as
default methods, leaving hv_train_network.py with only parser + main while
other architectures properly subclass NetworkTrainer. This broke the symmetry
and forced HV-specific imports (HYVideoDiffusionTransformer, VAE_VER,
text_encoder_module, ...) into the shared base file.

Introduce HunyuanVideoNetworkTrainer in hv_train_network.py and make the
corresponding methods in NetworkTrainer abstract hooks (NotImplementedError).
The base class is now architecture-agnostic, and HV becomes consistent with
Wan/Qwen/FLUX/FramePack/Z-Image/HV1.5/Kandinsky5 subclasses.

Completes the outstanding Step 1 item from references/refactoring_plan.md.

* Update overview.md: simplify and reflect post-refactoring structure

Reduce from ~100 lines to ~45 lines by removing details derivable from
code. Add training/ directory, Kandinsky 5 architecture, and tree-style
directory layout. Consolidate workflow into a clear 3-step pipeline.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(train): log Dynamo cache-entry count for --compile recompile visibility

Adds a `dynamo/cache_entries` scalar to the per-step TensorBoard/W&B logs
whenever `--compile` is on. It uses the new public-enough Python API
`torch._dynamo.eval_frame._get_total_cache_entry_count` (landed post-2.12)
to count Dynamo cache entries across all `torch.compile`-wrapped block
forwards in the transformer, giving a live curve of recompile growth —
the actionable alternative to `TORCH_LOGS=recompiles` log scraping for
deciding whether `compile_cache_size_limit` is well-sized.

Key correctness point: the helper dedupes by `forward.__code__` because
Dynamo caches are keyed on code objects, not module instances. N copies
of the same block class share one cache entry; summing per-instance
would over-report by Nx (and FLUX.2 Klein 9B with ~56 same-class blocks
would inflate the number badly). Tests in this conversation verified
the dedup against models with 3 and 10 identical blocks.

Robustness: the API import is wrapped in try/except so the helper
returns 0 on pre-2.13 builds, and per-block queries are isolated so a
single failed lookup cannot kill telemetry. The accelerator's
self-reference shim on the top-level transformer
(`transformer.__dict__["_orig_mod"] = transformer` at
hv_train_network.py:2304) is filtered out via an `orig is module`
guard so it doesn't contribute a spurious entry.

Gated on `args.compile` and on the scalar being >0, so non-compile
runs and pre-2.13 environments produce no new TensorBoard series.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Refactor: introduce extension seams on NetworkTrainer (for Self-Flow + future extensions) (kohya-ss#943)

* Add process_batch() extension seam (no-op: behavior unchanged)

Extracts the inner per-step computation (timestep sampling, call_dit, MSE loss)
out of _run_training_loop into a process_batch() method on NetworkTrainer.
Default implementation reproduces the existing vanilla flow-matching path bit-for-bit.

This is the first of the in-tree extension seams. Marked internal-only
(no API stability guarantees) so we can refactor freely. Subclasses that need
custom training logic (e.g. Self-Flow with dual-timestep scheduling) can
override process_batch() instead of patching the training loop body.

* Add lifecycle extension hooks (on_train_start / on_post_optimizer_step / on_sample_images / on_post_save)

All four are no-op defaults so behavior is unchanged. Hook points:

- on_train_start: at the top of _run_training_loop, after accelerator.prepare
  has happened and before any training iteration. For initializing extension
  state that depends on prepared models (EMA copies, schedulers, forward hooks).

- on_post_optimizer_step: right after optimizer.step / lr_scheduler.step /
  zero_grad, with sync_gradients passed through. For EMA updates and
  per-step bookkeeping.

- on_sample_images: an around-hook wrapping the three self.sample_images()
  call sites (sample_at_first / mid-step / end-of-epoch). Default invokes
  the inner sample_fn() directly. Override to swap in EMA weights for
  sampling.

- on_post_save: called at the end of save_model() after the main checkpoint
  has been written. For saving companion files (EMA, projection heads)
  alongside.

All marked internal-only (no API stability guarantees).

* Add contributor extension hooks (extra_trainable_params / extra_metadata / extra_step_logs)

These complete Phase 1 of the in-tree extension seam set. All three are
no-op-default contributors: they return values that the trainer merges
into existing structures, so behavior is unchanged.

- extra_trainable_params: called inside _build_optimizer_and_dataloader
  right after network.prepare_optimizer_params, before get_optimizer.
  Subclasses can fold extra modules' parameters (e.g. a representation
  projection head) into the optimizer's param-group list. Threaded
  'transformer' through the private _build_optimizer_and_dataloader
  signature so subclasses can size their modules from transformer
  config (hidden_size etc.).

- extra_metadata: returns a dict merged into the safetensors metadata
  built at training-loop start. For ss_self_flow_* style keys.

- extra_step_logs: returns a dict merged into the per-step log payload
  just before accelerator.log. For self_flow/* and grad/* metrics.

All three internal-only (no API stability guarantees).

* Phase 2: convert call_dit return to DiTOutput dataclass + accept **kwargs

The return type of NetworkTrainer.call_dit was a (pred, target) tuple.
This change makes it a typed DiTOutput dataclass with optional 'features'
and an 'extra' dict escape hatch, and threads **kwargs through the
signature so extension subclasses can pass additional conditioning
(e.g. per_token_timesteps for Self-Flow) or request side outputs
(e.g. hidden_features) without further signature churn.

DiTOutput lives in training/trainer_base.py and is re-exported from
hv_train_network so the per-architecture trainers can import it via
the same path they use for NetworkTrainer.

Updated:
- training/trainer_base.py: DiTOutput dataclass, abstract call_dit
  signature with **kwargs and DiTOutput return type, default
  process_batch impl uses output.pred / output.target
- All 9 architecture call_dit overrides: hv, hv_1_5, wan (call_dit and
  inner _call_dit), flux_kontext, flux_2, fpack, kandinsky5,
  qwen_image, zimage
- Two finetune entry scripts that call self.call_dit directly:
  qwen_image_train.py, zimage_train.py

Internal extension point — no API stability guarantees.

* Phase 3: add Self-Flow skeleton (flux_2_train_network_self_flow.py)

Skeleton subclass of Flux2NetworkTrainer demonstrating how the Phase 1-2
extension seams accommodate Self-Flow. Each seam is overridden as a
shape-only stub: argument validation in handle_model_specific_args,
rep_proj construction in extra_trainable_params, EMA / scheduler /
forward-hook setup in on_train_start, dual-timestep loss in process_batch,
EMA update in on_post_optimizer_step, EMA-weight sampling in
on_sample_images, companion-file saving in on_post_save, plus
extra_metadata and extra_step_logs.

Algorithmic logic is left as NotImplementedError + TODO comments
referencing PR kohya-ss#913, so the file is not yet runnable. The intent is to
share this with rockerBOO as the concrete extension surface he'd build
Self-Flow on after the dev branch lands on main, and let him judge
whether the seams cover everything Self-Flow needs (and surface gaps
before the seams calcify).

Open question for rockerBOO surfaced in call_dit's docstring:
per-token-timestep conditioning requires support inside the FLUX.2
transformer itself, which this skeleton does not add — that's a per-arch
follow-up, not a base-class seam concern.

* Add on_before_sample_images and on_after_sample_images

* Add docstrings

* Update self flow example

* Refactor: relocate sample_images hooks to _do_sample

kohya-ss#945 added on_before/on_after_sample_images calls inside sample_images,
which required adding `network` to sample_images's signature solely so the
hooks could receive it. This broke qwen_image_train.py / zimage_train.py
which call sample_images without `network`, and was a smelly addition for
fine-tuning trainers that don't have a network at all.

Move the hook calls to _do_sample (where `network` is already in scope),
restoring sample_images's original signature. Add try/finally so the
after-hook runs on exception (necessary for Self-Flow's EMA weight swap).
Add should_sample_images guard at _do_sample's entry so hooks don't fire
on epoch-end when sampling is skipped.

Hook signatures themselves unchanged — Self-Flow skeleton override needs
no update.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* Address review: drop DiTOutput.features, add force_sync_upload to on_post_save

Two small review responses on PR kohya-ss#943:

- DiTOutput.features removed. The field was extension-only (no vanilla code
  path constructs it), and rockerBOO correctly pointed out that anything
  beyond pred/target should go through the `extra` dict so the base return
  type stays minimal. Self-Flow skeleton's call_dit docstring updated to
  reference DiTOutput.extra["features"] accordingly.

- on_post_save now receives force_sync_upload alongside the existing args.
  Subclasses uploading companion files (e.g. Self-Flow's EMA / projection
  heads) need this to mirror the main checkpoint's HuggingFace sync-upload
  behaviour. Self-Flow skeleton override updated.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* Add on_transformer_loaded post-load hook

New extension seam called from trainer_base immediately after
self.load_transformer(...) returns, before the model goes into eval mode
and before accelerator.prepare wraps it. Default no-op.

Intended use case (raised by rockerBOO): registering register_forward_hook
on the raw transformer for feature extraction without needing to override
load_transformer wholesale. Self-Flow skeleton's forward-hook registration
moves here from on_train_start step 6, since attaching pre-prepare keeps
captured outputs aligned with the unwrapped block indices users supply
via --student_feature_layer / --teacher_feature_layer.

Scoped to the transformer for now; text encoders are loaded per-arch in
varied ways (often via transformers directly, only at sample-prompt prep
time), and there is no concrete VAE-hook need yet.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* Extract compute_loss; process_batch returns (loss, loss_metrics)

Splits the loss formulation out of process_batch into a new compute_loss
seam that owns the full loss computation including weighting and reduction.
Subclasses can now override either:

- process_batch — to change what gets fed to the model (Self-Flow's
  dual-timestep dance, etc.) while keeping the standard loss form.
- compute_loss — to swap the loss formulation while keeping the standard
  data flow.

compute_loss takes responsibility for the full computation rather than just
the per-element MSE: weighting is tightly coupled to the vanilla flow-matching
loss form, and Self-Flow-style losses will swap the entire formulation, so a
finer-grained split would force them to re-express weighting in their own
terms anyway.

process_batch's return type is now tuple[Tensor, dict[str, float]]. The
metrics dict is merged into the per-step log payload alongside extra_step_logs
(separate concerns: loss decomposition vs. arbitrary subclass logs). Default
loss_metrics is empty; subclasses populate as needed (e.g. {"loss/gen": ...,
"loss/rep": ...} for Self-Flow).

Vanilla flow-matching loss order/values unchanged.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* format: by ruff

* Pass transformer to on_post_optimizer_step and on_post_save

Addresses PR kohya-ss#943 review: subclasses doing non-network (full fine-tuning)
bookkeeping or transformer-side EMA/companion saving had no way to reach
the transformer without stashing a reference in on_train_start. Both hooks
now receive it directly, matching the symmetry of on_train_start and
on_before/after_sample_images. Self-Flow override updated accordingly.

---------

Co-authored-by: rockerBOO <rockerboo@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

* fix: restore block swap after no-grad forwards

Restore block-swap layout after prior teacher no-grad forwards, harden sample generation cleanup, and add CUDA/manual smoke coverage for FLUX.2 block swap with EMA/DoRA LoRA.

* fix(attention)+test: graceful xformers degradation + peak-memory harness

Two changes that surfaced during the xzuyn-optimizations round-2 evaluation
and stand on their own merit independent of the rejected perf branch.

1) modules/attention.py: relax xformers import gate from
     except ImportError -> except (ImportError, OSError)
   The original ImportError-only clause does not catch OSError raised when
   xformers is installed but its native extension (mslk.so / _C.so) fails to
   dlopen against a freshly-rebuilt torch ABI. Without this, a stale
   daily-build artifact takes down the entire blissful-tuner import chain.
   Treating "broken extension" the same as "extension not available" matches
   what every other optional-backend gate in this file already does.

2) tests/manual/measure_flux2_block_peak_memory.py: standalone harness for
   torch.cuda.max_memory_allocated() comparison on Klein-9B-shaped
   DoubleStreamBlock + SingleStreamBlock forward+backward, with optional
   DoRA-LoRA wrapping. Synthetic random-init blocks at the right shape capture
   the same activation profile as a real checkpoint, so no 8.5 GB safetensors
   file is required and the live training config isn't disturbed. Used here
   to verify the round-2 perf branch produced 0 MB of per-block savings at
   production shape.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* doc(claude.md): record xzuyn-optimizations round 2 null result + LoRA bypass class

Round 2 (2026-05-20) again returned a null perf result: 0 MB per-block peak
savings at Klein-9B production shape. Documents the cheaper per-block
peak-memory check (`tests/manual/measure_flux2_block_peak_memory.py`) as a
pre-A/B filter, and adds the LoRA-bypass-via-weight-slicing pattern as a
distinct, recurring failure class that future perf-port reviews should check
for first.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* Fix: latent bugs surfaced by Copilot review on PR kohya-ss#950

- trainer_base._build_network: pass imported network_module object to
  convert_weight_keys instead of the args.network_module string. The
  hook's type annotation already declared a module; current overrides
  ignore the argument so it had not manifested.
- parser_common._int_or_float: use int(float_value) instead of
  int(value) so inputs like "1.0"/"10.0" (common from TOML) don't
  fall through to a misleading 'not an int or float' error.
- flux_2_train_network_self_flow: initialise _saved_student_state in
  __init__ and guard on_after_sample_images with the same self_flow /
  ema_lora_state checks as on_before_sample_images so sampling does
  not AttributeError when Self-Flow is off.

* doc(claude.md): record xzuyn-optimizations round 3 (Triton kernels) rejection

Round 3 (2026-05-21) introduces three custom Triton kernels (RopeTriton,
ApplyRopeTriton, AdaLNTriton). Rejected on numerics alone, no measurement
run — AdaLNTriton's self-reported bf16 backward max diff of 0.5 (file
header table in triton_kernels.py) is the standalone disqualifier given
AdaLN compounds across 32 Klein-9B blocks per forward.

Other notable observations folded into the entry:
  - kernels are default-on with no opt-in flag
  - compile=true would hit @triton.jit boundaries 32x per forward (same
    Inductor-fusion-barrier trap as last cycle's Liger 19% regression)
  - xzuyn's own FusedGateResidualTriton add/remove oscillation ended with
    commit message "peak memory ended up being worse" - corroborates
    round-2's null measurement
  - no rank_dropout-style convergence signal this round; tempering the
    round-2 "fork-as-confirmatory-signal" framing accordingly
  - LoRA-bypass-via-weight-slicing pattern persists in commits 1d50b0b
    and 9f00186; strengthened the section to note recurrence across all
    three review cycles

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* Bump version 0.3.0 (kohya-ss#951)

* doc: Update README. Document large-scale internal refactoring for improved code quality and maintainability

* chore: bump version to 0.3.0 in pyproject.toml

* doc(plans): record refactor-landed migration strategy + agent analyses

Upstream kohya-ss/musubi-tuner refactor merged to main (PR kohya-ss#950, v0.3.0).
Decision recorded: MERGE upstream into blissful + a post-merge dead-weight
cleanup commit, NOT a green-field rebuild. Architecture parity (all 9 model
families ship natively upstream, blissful ahead on 5) makes this a
feature-port, not a model-port.

Measured surface: 1304 ahead / 45 behind; virtual merge = 40 conflicts,
32 of them keep-ours (blissful is ahead). Hard work isolated to the
hv_train_network.py structural split (own plan) + the dataset/ refactor.
Effort ~14-19h. Verified the 2026-05-21 masked-loss plan's trainer_base
seam line numbers still hold on upstream/main (dev->main was a fast-forward).

Adds (force-added past the docs/plans/ .gitignore, matching prior plan docs):
- 2026-05-23-refactor-landed-migration-strategy.md  decision record + roadmap
- ...-A-conflict-ledger.md     full 40-conflict file-by-file triage
- ...-B-feature-inventory.md   must-keep training features + arch parity
- ...-C-deadweight-audit.md    import-graph; ~60K LOC safe-delete + gotchas

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* doc(plans): incorporate Ampere review — revise estimate, fix counts, name seam gaps

Verified every load-bearing claim against upstream/main (78acafb) + HEAD,
then patched the three 2026-05-23 migration docs:

- Estimate 14-19h -> 18-25h (bottom-up; trainer port + dataset split bigger
  than first scoped). Direction unchanged: merge, do not rebuild.
- Conflict-ledger category counts corrected: 22 BLISSFUL-AHEAD / 10 TRIVIAL
  (= 32 keep-ours), not 21/11 / "31". Re-tallied from the data rows.
- Dataset split is FIVE new files (architectures/bucket/cache_io/datasources/
  media_utils), not two; image_video_dataset re-home raised to ~2-2.5h with the
  full must-preserve feature list (caption_directory, multi-dot caption filter,
  duplicate-basename guard, mask fallback, cache mask metadata, Qwen layered/
  multiple_target, Z-Image/FLUX.2 cache-mask conventions).
- New "Trainer seam — what does NOT come for free" section (all verified):
  full-FT QwenImageTrainer/ZImageTrainer run own train() loops (no process_batch
  inheritance); call_dit returns a tuple while upstream uses DiTOutput dataclass;
  restore_block_swap helper is blissful-only and must be ported; parser args /
  mask validation / disabled-mask warning are setup-time, not in process_batch.
- config_utils rename de-risked: both sides already carry the deprecated-key
  shim; broadened the straggler grep to catch qwen_image_edit_control_resolution.
- Qwen guardrails: protect padded-text/varlen cu_seqlens attention region; the
  layered one-control guard is already in HEAD (no pending cherry-pick).
- Tier B flagged training-import-safe but NOT test-safe (delete generate-script
  tests in the same commit).
- Added the two targeted pytest gates as the first acceptance ladder.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(lora,mask-loss): post-merge review fixes — LoKr factor loss + block-swap restore-in-finally

Follow-up to the v0.3.0 merge (35c5ecc), addressing two issues found in review.

fix(lora): create_network_from_weights silently discarded an explicit module_kwargs.
  `module_kwargs = kwargs.pop("module_kwargs", None)` ran unconditionally while module_kwargs
  is a named parameter (upstream signature), so a caller passing it as the named kwarg got it
  popped to None. LoKr's create_arch_network_from_weights resolves the factor (_resolve_factor
  over the lokr_factor buffer / ss_lokr_factor metadata) and forwards {"factor": N} here → it
  was dropped → LoKr rebuilt with factor=-1 (automatic factorization) → lokr_w1/lokr_w2 shape
  mismatch on reload via --base_weights / --dim_from_weights / wan_generate_video. Now mirrors
  the guard already on the twin create_network; also guards module_class (previously survived
  only by accident). +test_lokr_factor_roundtrip.py: explicit module_kwargs is not discarded.

fix(mask-loss): move the block-swap restore into `finally` in masked_process_batch.
  A prior-teacher call_dit raising mid-forward would skip restore_block_swap_after_no_grad_forward,
  stranding swapped blocks on CPU → next training forward crashes "mat2 is on cpu". Now runs
  regardless of whether the teacher forward raises. +test_masked_process_batch.py: restore fires
  when the teacher forward raises.

doc(claude.md): record two merge tripwires — compute_loss_weighting_for_sd3 must keep its
  n_dim= param (W14; guarded by test_loss_weighting_ndim, re-restore on every upstream merge),
  and the post-merge `git add -u` + grep-verify index-staleness check plus the splice-bug class.

Targeted regression bundle green (103 passed); ruff check + format clean. GPU WAN numerical
parity gate remains deferred — owner's call before release.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* doc(claude.md): WAN fp8 flag-pairing + block-swap/offload mutual-exclusion gotchas

From the v0.3.0 parity-gate investigation:
- --fp8_base ALONE is a blunt full-model fp8 cast (norms/modulation included) that
  WAN 2.2's TP-03 guard correctly rejects. The working WAN fp8 path is
  --fp8_base --fp8_scaled TOGETHER (per-layer scaled fp8, excludes norm/modulation).
  By design, byte-identical across the merge — not a regression.
- --blocks_to_swap and --offload_inactive_dit are mutually exclusive (WAN asserts).
Both cost a debug cycle during the GPU parity gate.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(lint): resolve ruff F405/F821 + format debt from the v0.3.0 dataset split

CI Ruff Lint (PR #8) failed. Two classes, both dataset-split artifacts:
- F405 (x5, image_video_dataset.py): glob_images + resize_mask_to_bucket were used
  locally but arrived via `from media_utils import *`; the merge added the explicit
  import for resize_image_to_bucket only. Added the missing two. Not a runtime bug —
  the star import resolved them; ruff just couldn't verify (full suite was green).
- F821 (x4, cache_io.py): List/Sequence/BaseDataset referenced in string annotations
  (from __future__ import annotations, so never evaluated → no runtime error) but not
  importable. Added List/Sequence to the typing import; BaseDataset to the TYPE_CHECKING
  block (a runtime import would be circular: image_video_dataset imports cache_io).
- ruff format: 7 files the merge left format-unclean (the 4 dataset-split modules, the
  z-image LoRA converter, 2 tests). Format-only, no logic change.

ruff check + ruff format --check both clean locally; dataset/mask/parity bundle green
(30 passed); all 5 touched modules import.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Kohya S. <52813779+kohya-ss@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Christopher <christopher.bourez@gmail.com>
Co-authored-by: Obsxrver <206303661+obsxrver@users.noreply.github.com>
Co-authored-by: Ben Lambert <blambert@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rockerBOO <rockerboo@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kohya-ss