Skip to content

Add types and MLX compatibility for cli generation#93

Merged
Saibo-creator merged 3 commits intoepfl-dlab:mainfrom
nathanrchn:types_mlx
Sep 11, 2024
Merged

Add types and MLX compatibility for cli generation#93
Saibo-creator merged 3 commits intoepfl-dlab:mainfrom
nathanrchn:types_mlx

Conversation

@nathanrchn
Copy link
Copy Markdown
Contributor

  1. Type Annotations

    • Added type hints throughout the codebase
    • Improves code readability and helps developers understand the library's structure and API
  2. MLX Compatibility

    • Integrated MLX support for faster generation
    • New CLI argument: --use-mlx to enable MLX acceleration
    • Note: Quantization arguments are not yet compatible with MLX (--use_4bit and --use_8bit)
    • Note: I am still waiting to be merged on the mlx_lm repo, so the url to install mlx_lm will (maybe) change.

Usage Example

To use MLX acceleration:

transformers-cfg-cli generate \
    -m "mlx-community/Phi-3-mini-4k-instruct-4bit" \
    -g "examples/grammars/json.ebnf" \
    -p "This is a valid json string for http request:" \
    --use_mlx \
    --max_new_tokens 60 \
    --repetition_penalty 1.1
# {"name":"John","age":30,"car":null}

@Saibo-creator Saibo-creator merged commit 36e263e into epfl-dlab:main Sep 11, 2024
@nathanrchn nathanrchn deleted the types_mlx branch September 12, 2024 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants