Fix QAT + LoRA fast path, add tests #3307
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andrewor14
commented
Sep 15, 2025
| # TODO: there are bad interactions across tests right now, need to figure out | ||
| # how to disable model caching before re-enabling this test | ||
| @pytest.mark.parametrize("qat_scheme", ["fp8-int4", "fp8-fp8"]) | ||
| def _test_full_model_fake_quantize(qat_scheme: bool): |
Contributor
Author
There was a problem hiding this comment.
@danielhanchen seems like there's some model state being cached across tests, is there an easy way to clear the cache in between the tests or disable the caching somehow?
Contributor
There was a problem hiding this comment.
Do you mean the overridden functions?
Contributor
Author
There was a problem hiding this comment.
like if I run the lora test by itself, then I get the right peft model, but if I re-enable this full finetuning test and run the whole test file at once, then the lora test no longer gives me a peft model
andrewor14
force-pushed
the
fix-qat-lora
branch
2 times, most recently
from
3b6bcf8 to
0eaad5e
Compare
**Summary:** The existing QAT + LoRA path only applied fake quantization to the original slow path, but the default is the fast path that calls unsloth's fast LoRA primitives. This commit integrates fake quantization into these fast primitives as well, and add unit tests to assert that fake quantization is actually taking place. **Test Plan:** Unit tests: ``` pytest tests/utils/test_qat.py ``` End-to-end test: https://gist.github.com/andrewor14/6360dd69b5784c71c46e80c14f53e6b6 Full fine-tuning Llama3.1-8B with and without QAT + LoRA on yahma/alpaca-cleaned for 1 epoch: - Batch size = 8 (no grad accum) - Learning rate = 2e-4 - Quantization scheme = int4 weight only (with bf16 activations) Wikitext perplexity: - Baseline = int4 quantized model finetuned without QAT - QAT int4 quantized model (with this PR) achieved 33% lower perplexity than the int4 baseline - QAT int4 quantized model without this PR was worse than the int4 baseline ``` ==> unsloth_model_lora_baseline_output/lm_eval_float.log <== | | |none | 0|word_perplexity|↓ |7.5551|± | N/A| ==> unsloth_model_lora_baseline_output/lm_eval_quantized.log <== | | |none | 0|word_perplexity|↓ |8.7655|± | N/A| ==> unsloth_model_lora_qat_int4_output/lm_eval_quantized.log <== | | |none | 0|word_perplexity|↓ |8.3548|± | N/A| ```
andrewor14
force-pushed
the
fix-qat-lora
branch
from
0eaad5e to
56f2087
Compare
Contributor
|
Nice work! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: The existing QAT + LoRA path only applied fake quantization to the original slow path, but the default is the fast path that calls unsloth's fast LoRA primitives. This commit integrates fake quantization into these fast primitives as well, and add unit tests to assert that fake quantization is actually taking place.
Test Plan:
Unit tests:
End-to-end test: https://gist.github.com/andrewor14/6360dd69b5784c71c46e80c14f53e6b6
Full fine-tuning Llama3.1-8B with and without QAT + LoRA on yahma/alpaca-cleaned for 1 epoch:
Wikitext perplexity: