[misc][core] lazy import outlines #7831

youkaichao · 2024-08-24T05:20:50Z

temp fix for #4193

for users who don't use guided decoding but use slurm cluster, this makes their lives eaiser.

github-actions · 2024-08-24T05:21:04Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

Comment /ready on the PR
Add ready label to the PR
Enable auto-merge.

🚀

simon-mo · 2024-08-24T05:26:30Z

Nice this is good fix for now! But the users will still have problem with the actually usage :(

youkaichao · 2024-08-24T05:28:00Z

yep this is a temp fix, because I heard many people are suffering from this problem but they don't use outlines at all.

for people who really want guided decoding and suffer from the problem, they can use --guided-decoding-backend=lm-format-enforcer :)

youkaichao · 2024-08-24T05:28:33Z

I'm waiting for users' confirmation to see if it works.

chujiezheng · 2024-08-24T07:36:22Z

Tested on slurm and this pr works well for me

alex2awesome · 2024-08-29T07:59:35Z

when will this be released? I'm unable to build from source on my HPC server

youkaichao · 2024-08-29T16:14:52Z

@alex2awesome you don't need to wait for the release, we have per-commit wheel released. see

https://docs.vllm.ai/en/latest/getting_started/installation.html

alex2awesome · 2024-08-29T20:16:23Z

That's good to know!! unfortunately, after installing the nightly build, I'm still getting this error. Is there a way to delete/refresh the database?

  File "/project/jonmay_231/spangher/Projects/conditional-information-retrieval/source_summaries/data_vllm_70b.py", line 19, in <module>
    from vllm import LLM,  SamplingParams
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/vllm/__init__.py", line 6, in <module>
    from vllm.entrypoints.llm import LLM
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 13, in <module>
    from vllm.model_executor.guided_decoding import (
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/vllm/model_executor/guided_decoding/__init__.py", line 8, in <module>
    from vllm.model_executor.guided_decoding.outlines_decoding import (
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py", line 15, in <module>
    from vllm.model_executor.guided_decoding.outlines_logits_processors import (
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/vllm/model_executor/guided_decoding/outlines_logits_processors.py", line 25, in <module>
    from outlines import grammars
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/outlines/__init__.py", line 2, in <module>
    import outlines.generate
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/outlines/generate/__init__.py", line 2, in <module>
    from .cfg import cfg
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/outlines/generate/cfg.py", line 3, in <module>
    from outlines.fsm.guide import CFGGuide
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/outlines/fsm/guide.py", line 109, in <module>
    def create_states_mapping(
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/outlines/caching.py", line 93, in decorator
    memory = get_cache()
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/outlines/caching.py", line 65, in get_cache
    memory["__version__"] = outlines_version
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/diskcache/core.py", line 823, in __setitem__
    self.set(key, value, retry=True)
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/diskcache/core.py", line 806, in set
    self._row_update(rowid, now, columns)
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/diskcache/core.py", line 828, in _row_update
    sql(
sqlite3.DatabaseError: database disk image is malformed```

youkaichao · 2024-08-29T21:18:43Z

File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/vllm/model_executor/guided_decoding/init.py", line 8, in
from vllm.model_executor.guided_decoding.outlines_decoding import (

your installation might be wrong. if you have the latest commit installed, line 8 should not be this one.

see

vllm/vllm/model_executor/guided_decoding/__init__.py

Line 8 in 257afc3

from vllm.sampling_params import LogitsProcessor

alex2awesome · 2024-08-30T00:18:54Z

Ahh thanks @youkaichao — dumb error on my part, I just copy/pasted the instructions in the docs. The right version to use for anyone coming here is:

export VLLM_VERSION=0.5.5
pip install https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-${VLLM_VERSION}-cp38-abi3-manylinux1_x86_64.whl```

gustavosm · 2024-08-30T13:50:50Z

Hello, guys!!
I'm facing same problem as @alex2awesome, but I don't know what I can do here, because I'm running a golang app to start a docker container for vllm. So, I'm not sure how to run this pip install mentioned above, since my container drops as soon as it tries to run.
I have tried vllm 0.5.5 version, but no success.
Any suggestion?
The error message I'm getting is exactly the same as @alex2awesome posted few comments above.

Something that might worth mention is: the problem only occurs if I run the app with my linux user... I log in with another user, everything works fine

LucWeber · 2024-08-30T20:03:29Z

Hello, guys!! I'm facing same problem as @alex2awesome, but I don't know what I can do here, because I'm running a golang app to start a docker container for vllm. So, I'm not sure how to run this pip install mentioned above, since my container drops as soon as it tries to run. I have tried vllm 0.5.5 version, but no success. Any suggestion? The error message I'm getting is exactly the same as @alex2awesome posted few comments above.

Something that might worth mention is: the problem only occurs if I run the app with my linux user... I log in with another user, everything works fine

As I understand it the fix is not yet in 0.5.5. You will have to install from source

alex2awesome · 2024-09-02T23:46:49Z

Hi @youkaichao , has the fix been implemented with the OpenAI-compatible inference engine yet?

I've tested and am OK with loading the model in python, as in: https://docs.vllm.ai/en/latest/getting_started/quickstart.html

but when I try to launch an inference engine using:

python -m vllm.entrypoints.openai.api_server \
    --model NousResearch/Meta-Llama-3-70B-Instruct \
    --dtype float16 \
    --tensor-parallel-size $NUM_GPUS \
    --api-key token-abc123 \
    --enforce-eager &

I get the same errors:

  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/vllm/model_executor/guided_decoding/outlines_decoding.py", line 15, in <module>
    from vllm.model_executor.guided_decoding.outlines_logits_processors import (
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/vllm/model_executor/guided_decoding/outlines_logits_processors.py", line 25, in <module>
    from outlines import grammars
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/outlines/__init__.py", line 2, in <module>
    import outlines.generate
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/outlines/generate/__init__.py", line 2, in <module>
    from .cfg import cfg
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/outlines/generate/cfg.py", line 3, in <module>
    from outlines.fsm.guide import CFGGuide
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/outlines/fsm/guide.py", line 109, in <module>
    def create_states_mapping(
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/outlines/caching.py", line 93, in decorator
    memory = get_cache()
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/outlines/caching.py", line 65, in get_cache
    memory["__version__"] = outlines_version
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/diskcache/core.py", line 823, in __setitem__
    self.set(key, value, retry=True)
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/diskcache/core.py", line 806, in set
    self._row_update(rowid, now, columns)
  File "/home1/spangher/miniconda3/envs/vllm-py310/lib/python3.10/site-packages/diskcache/core.py", line 828, in _row_update
    sql(
sqlite3.DatabaseError: database disk image is malformed```

youkaichao · 2024-09-03T00:36:55Z

@alex2awesome it should also work for the api server.

your stack trace is incomplete, and it is unclear if you use guided decoding or not.

Signed-off-by: LeiWang1999 <[email protected]>

youkaichao added 3 commits August 23, 2024 22:09

lazy import outlines

7d725cd

add tests

54991b8

add tests

fcdde91

simon-mo approved these changes Aug 24, 2024

View reviewed changes

DarkLight1337 approved these changes Aug 24, 2024

View reviewed changes

youkaichao added 5 commits August 24, 2024 00:22

lazy import in lm-format-enforcer

31c2c44

add lm-format-enforcer test

fe1f35c

fix tests

26a1ce6

add comments

79d1a80

add comments

1818eec

youkaichao merged commit 7d9ffa2 into vllm-project:main Aug 24, 2024

youkaichao deleted the lazy_outlines branch August 24, 2024 07:51

youkaichao mentioned this pull request Aug 24, 2024

[Bug]: Disk I/O Error when using tools due to shared outlines cache database #4193

Closed

youkaichao mentioned this pull request Sep 2, 2024

[Bug]: vllm0.4.3 guided decoding #8057

Closed

1 task

e-tornike mentioned this pull request Sep 16, 2024

Running multiple processes on a shared outlines cache database EleutherAI/lm-evaluation-harness#2306

Open

youkaichao mentioned this pull request Oct 1, 2024

[Bug]: Bus error (core dumped) #8974

Closed

1 task

Alvant pushed a commit to compressa-ai/vllm that referenced this pull request Oct 26, 2024

[misc][core] lazy import outlines (vllm-project#7831)

16e2c8e

LeeSureman mentioned this pull request Nov 24, 2024

[Bug] disk cache io error when simultaneously loading lots of sglang offline engine sgl-project/sglang#2090

Open

5 tasks

LeiWang1999 pushed a commit to LeiWang1999/vllm-bitblas that referenced this pull request Mar 26, 2025

[misc][core] lazy import outlines (vllm-project#7831)

9550eaa

Signed-off-by: LeiWang1999 <[email protected]>

Uh oh!

[misc][core] lazy import outlines #7831

[misc][core] lazy import outlines #7831

Uh oh!

Conversation

youkaichao commented Aug 24, 2024

Uh oh!

github-actions bot commented Aug 24, 2024

Uh oh!

simon-mo commented Aug 24, 2024

Uh oh!

youkaichao commented Aug 24, 2024

Uh oh!

youkaichao commented Aug 24, 2024

Uh oh!

chujiezheng commented Aug 24, 2024

Uh oh!

alex2awesome commented Aug 29, 2024

Uh oh!

youkaichao commented Aug 29, 2024

Uh oh!

alex2awesome commented Aug 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

youkaichao commented Aug 29, 2024

Uh oh!

alex2awesome commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gustavosm commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LucWeber commented Aug 30, 2024

Uh oh!

alex2awesome commented Sep 2, 2024

Uh oh!

youkaichao commented Sep 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

alex2awesome commented Aug 29, 2024 •

edited

Loading

alex2awesome commented Aug 30, 2024 •

edited

Loading

gustavosm commented Aug 30, 2024 •

edited

Loading