Skip to content

Conversation

@bjmsong
Copy link
Contributor

@bjmsong bjmsong commented Feb 21, 2025

Motivation

Relate to #3571, some AWQ models are incompatible with marlin kernels.

Modifications

Use unoptimized kernel if the models are incompatible with marlin kernels.

test script

python examples/runtime/engine/offline_batch_inference.py --model=${DeepSeek-V2-Lite-Chat-AWQ} --trust-remote-code

refer to this PR

Checklist

@merrymercy merrymercy requested a review from HaiShaw as a code owner March 3, 2025 08:12
@github-actions github-actions bot closed this May 30, 2025
@github-actions
Copy link
Contributor

This pull request has been automatically closed due to inactivity. Please feel free to reopen it if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant