Fix stablelm

[esmeetu]

Stabilityai has uploaded transformers implementation. https://huggingface.co/stabilityai/stablelm-3b-4e1t/discussions/10

from vllm import LLM, SamplingParams

# Sample prompts.
prompts = [
    "Hello, my name is"
]
# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.0, max_tokens=64)

# Create an LLM.
llm = LLM(model="stabilityai/stablelm-3b-4e1t", dtype="half", tensor_parallel_size=1, enforce_eager=True, trust_remote_code=True)
# Generate texts from the prompts. The output is a list of RequestOutput objects
# that contain the prompt, generated text, and other information.
outputs = llm.generate(prompts, sampling_params)
# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Ouput:

Prompt: 'Hello, my name is', Generated text: " and I'm writing you today to learn more about the 2018 Ford F-150 XLT SuperCrew 5.5' Box 4WD listed for $39,995.00. I live in the area and I would like to hear back from you soon and learn more about this vehicle. Please call me at at"

[WoosukKwon]

@esmeetu Should we replace the old version with this? Do we have a reason to keep the old one?

[simon-mo]

AFIAK the old version is needed for older huggingface version?

[WoosukKwon]

@simon-mo We already updated our transformers version to >=4.38.0 so it should be fine.

[esmeetu]

@WoosukKwon There only have small configuration changes on model and I think we can keep the backwards compatibility for a while.

[WoosukKwon]

@esmeetu Could you explain more about the backward compatibility? Do you mean the situation where the user uses an older version of transformers?

[esmeetu]

@esmeetu Could you explain more about the backward compatibility? Do you mean the situation where the user uses an older version of transformers?

No, this will provide a more kindly user experience for those (including me) who just use a local offline old repo before update. Otherwise when upgrading vLLM, it will throw an error and let them confused since they don't know that repo update information and think this is must an issue about vLLM upgrade.🤯

[WoosukKwon]

@esmeetu Got it. Thanks for the explanation!