[FEATURE] Enables /score endpoint for embedding models #12846

gmarinho2 · 2025-02-06T18:26:40Z

Enables /score endpoint for all embedding models via API.

Issue: [FEATURE] Enables offline /score for embedding models (2/2)

Signed-off-by: Gabriel Marinho <[email protected]>

github-actions · 2025-02-06T18:26:51Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: Gabriel Marinho <[email protected]>

maxdebayser · 2025-02-14T13:43:29Z

@DarkLight1337 and @K-Mistele , this PR is ready for review. The changes are:

Refactor OpenAIServingScores to enable the /score endpoint for embedding models. In this case the cosine similarity between the embedding pairs is returned.
Reuse OpenAIServingScores to also serve the /rerank endpoints. As a result of this refactoring, the /rerank endpoint now also support embedding models.
Improve /score unit tests to compare the results with the reference implementation (sentence-transformers)
Add an embedding model to the /score unit tests
Update the documentation.

DarkLight1337

Thanks for adding this! Some initial comments.

vllm/entrypoints/openai/serving_score.py

tests/entrypoints/openai/test_rerank.py

vllm/entrypoints/openai/serving_score.py

vllm/entrypoints/openai/api_server.py

Signed-off-by: Gabriel Marinho <[email protected]>

DarkLight1337

Thanks, LGTM now. @K-Mistele can you take another pass as this as well?

Signed-off-by: Gabriel Marinho <[email protected]>

Signed-off-by: Max de Bayser <[email protected]>

gmarinho2 · 2025-02-20T12:41:56Z

@joerunde

K-Mistele · 2025-02-20T17:45:11Z

Thanks for the ping @DarkLight1337 - looks good to me.

DarkLight1337 · 2025-02-21T06:09:41Z

Since main is broken, I'm force-merging this - test_score and test_rerank passed prior to the latest merge.

…12846)

…12846) Signed-off-by: Louis Ulmer <[email protected]>

…12846)

gmarinho2 and others added 18 commits January 21, 2025 11:18

[FEATURE] Enables offline /score for embedding models

9b0ce65

Signed-off-by: Gabriel Marinho <[email protected]>

Merge branch 'upstream_main'

74cd2dd

Signed-off-by: Gabriel Marinho <[email protected]>

Changes variable name and uses tuple instead of list.

590ab4d

Signed-off-by: Gabriel Marinho <[email protected]>

Separates scoring logic and makes small ajustments

f219024

Signed-off-by: Gabriel Marinho <[email protected]>

Separates scoring logic and makes small ajustments

47211e5

Signed-off-by: Gabriel Marinho <[email protected]>

Moves scoring functions declaration to llm class

1a89033

Signed-off-by: Gabriel Marinho <[email protected]>

Completes embedding_score function signature

db7919b

Signed-off-by: Gabriel Marinho <[email protected]>

Passes new parameters to self.encode() in embeddind_score

41b11be

Signed-off-by: Gabriel Marinho <[email protected]>

Adds type annotations for the parameters

b57e01a

Signed-off-by: Gabriel Marinho <[email protected]>

Minor adjustments

4956eae

Signed-off-by: Gabriel Marinho <[email protected]>

Minor adjustments in embedding_score

f8b8d8c

Signed-off-by: Gabriel Marinho <[email protected]>

trigger ci

cd835de

Signed-off-by: Gabriel Marinho <[email protected]>

trigger ci

0e95ead

Signed-off-by: Gabriel Marinho <[email protected]>

Merge branch 'vllm-project:main' into main

bba2ea6

Merge branch 'vllm-project:main' into main

635b8e8

first implementation of embedding scores via api

7851b44

Signed-off-by: Gabriel Marinho <[email protected]>

second version of api scoring

12383b2

Signed-off-by: Gabriel Marinho <[email protected]>

makes separate functions for cross-encoder score and embedding score

12ef932

Signed-off-by: Gabriel Marinho <[email protected]>

mergify bot added the frontend label Feb 6, 2025

fixes pre-commit errors

945280f

Signed-off-by: Gabriel Marinho <[email protected]>

gmarinho2 force-pushed the scoring-openai branch from 6d9077b to 945280f Compare February 7, 2025 13:49

adapts union sintax to python 3.9

053bdbb

Signed-off-by: Gabriel Marinho <[email protected]>

gmarinho2 force-pushed the scoring-openai branch 3 times, most recently from 3599577 to 8b0b41f Compare February 7, 2025 18:53

fixes alternating response bug

cfbc4b9

Signed-off-by: Gabriel Marinho <[email protected]>

gmarinho2 force-pushed the scoring-openai branch 2 times, most recently from c4762c7 to 2d44144 Compare February 7, 2025 23:53

fixes pre-commit errors

a9b3a0d

Signed-off-by: Gabriel Marinho <[email protected]>

gmarinho2 added 2 commits February 13, 2025 14:38

fixes union syntax

5c0495d

Signed-off-by: Gabriel Marinho <[email protected]>

adds documentation

7b2891b

Signed-off-by: Gabriel Marinho <[email protected]>

DarkLight1337 reviewed Feb 14, 2025

View reviewed changes

vllm/entrypoints/openai/serving_score.py Outdated Show resolved Hide resolved

vllm/entrypoints/openai/serving_score.py Outdated Show resolved Hide resolved

vllm/entrypoints/openai/serving_score.py Outdated Show resolved Hide resolved

K-Mistele reviewed Feb 14, 2025

View reviewed changes

tests/entrypoints/openai/test_rerank.py Show resolved Hide resolved

vllm/entrypoints/openai/serving_score.py Outdated Show resolved Hide resolved

vllm/entrypoints/openai/api_server.py Outdated Show resolved Hide resolved

gmarinho2 added 2 commits February 18, 2025 09:48

fixing mypy errors and refactoring

6218cc3

Signed-off-by: Gabriel Marinho <[email protected]>

Puts embedding score code in score_utils to avoid duplicated code

5b54495

Signed-off-by: Gabriel Marinho <[email protected]>

gmarinho2 force-pushed the scoring-openai branch from 6492787 to 5b54495 Compare February 18, 2025 14:18

changes variable name for clarity

17a2960

Signed-off-by: Gabriel Marinho <[email protected]>

DarkLight1337 approved these changes Feb 18, 2025

View reviewed changes

gmarinho2 and others added 3 commits February 18, 2025 12:51

refactoring serving_score

2a082f3

Signed-off-by: Gabriel Marinho <[email protected]>

factor out common code

077cbae

Signed-off-by: Max de Bayser <[email protected]>

remove extra code lines

1fa73a8

Signed-off-by: Max de Bayser <[email protected]>

Merge branch 'vllm-project:main' into scoring-openai

c9a7240

DarkLight1337 enabled auto-merge (squash) February 21, 2025 03:36

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Feb 21, 2025

simon-mo merged commit 1c3c975 into vllm-project:main Feb 21, 2025
40 of 58 checks passed

gmarinho2 deleted the scoring-openai branch February 21, 2025 18:17

Akshat-Tripathi pushed a commit to krai/vllm that referenced this pull request Mar 3, 2025

[FEATURE] Enables /score endpoint for embedding models (vllm-project#…

6cf9d59

…12846)

lulmer pushed a commit to lulmer/vllm that referenced this pull request Apr 7, 2025

[FEATURE] Enables /score endpoint for embedding models (vllm-project#…

c54a2cf

…12846) Signed-off-by: Louis Ulmer <[email protected]>

ckhordiasma mentioned this pull request Apr 17, 2025

[do not merge] pr test for nm changes into 2.20 red-hat-data-services/vllm#107

Closed

shreyankg pushed a commit to shreyankg/vllm that referenced this pull request May 3, 2025

[FEATURE] Enables /score endpoint for embedding models (vllm-project#…

39973a6

…12846)

Abirdcfly mentioned this pull request May 29, 2025

[Misc] remove unused jinaai_serving_reranking #18878

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[FEATURE] Enables /score endpoint for embedding models #12846

[FEATURE] Enables /score endpoint for embedding models #12846

Uh oh!

gmarinho2 commented Feb 6, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Feb 6, 2025

Uh oh!

maxdebayser commented Feb 14, 2025

Uh oh!

DarkLight1337 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 left a comment

Uh oh!

gmarinho2 commented Feb 20, 2025

Uh oh!

K-Mistele commented Feb 20, 2025

Uh oh!

DarkLight1337 commented Feb 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

[FEATURE] Enables /score endpoint for embedding models #12846

[FEATURE] Enables /score endpoint for embedding models #12846

Uh oh!

Conversation

gmarinho2 commented Feb 6, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 6, 2025

Uh oh!

maxdebayser commented Feb 14, 2025

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

gmarinho2 commented Feb 20, 2025

Uh oh!

K-Mistele commented Feb 20, 2025

Uh oh!

DarkLight1337 commented Feb 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gmarinho2 commented Feb 6, 2025 •

edited by github-actions bot

Loading