feat: Add GANQ quantizer #407
Closed
+778
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| lower_cutoff_idx = round(self.n * (1 - ratio) + 0.5) + 1 | ||
| lower_cutoff_values = sorted_W[:, lower_cutoff_idx + 1].unsqueeze(1) | ||
|
|
||
| outliers = (cutoff_values <= self.W) | (lower_cutoff_values >= self.W) |
There was a problem hiding this comment.
Bug: Outlier Split Index Error
The outlier_split method risks an IndexError. The lower_cutoff_idx calculation can produce an index such that lower_cutoff_idx + 1 exceeds self.n - 1, leading to out-of-bounds access on sorted_W for small self.n or certain ratio values.
| # pre_process=smash_config["pre_process"], | ||
| # full_rows=smash_config["full_rows"], | ||
| ) | ||
| gpt.free() |
There was a problem hiding this comment.
Bug: Quantizer Configuration System Broken
The gpt.fasterquant() method uses hardcoded default values for quantization parameters like weight_bits, max_epoch, and pre_process. The lines meant to pass these user-configured hyperparameters from smash_config are commented out, which makes the quantizer's configuration system non-functional.
|
This PR has been inactive for 10 days and is now marked as stale. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds the GANQ quantizer.
Related Issue
Fixes #391
Type of Change
How Has This Been Tested?
I have locally tested that the GANQ quantization works, however a lot of things need to be discussed with the team to write more formal tests. Will add.
Checklist
Additional Notes
This quantizer is not fully ready yet, I am creating a PR to get early feedback.
I will be fixing the issues I am getting with the pre-commit(like variable names, docstrings etc) soon. Some of those are failing and I had to use
--no-verify