Embeddings

[mwrothbe]

Nice project!

I'm interested in serving embeddings, and I see you have a placeholder api for it. Curious if that's on your short list to enable.

Thanks!

[SearchSavior]

Thanks! Actually yes.

Check out branch 1.0.6 (still WIP) to get an idea of how the new architecture could be extended- have you worked with embeddings in openvino before? I was thinking of using sentence transformers but the OV-classes from Optimum might give more flexibility? Haven't done a ton of work with embeddings before, so this would likely be something for 2.1 and end up in openai compatible /v1/embeddings.

Another pathway would be to use OpenVINO GenAI, but this would require some testing to validate.

[mwrothbe]

I've only used sentence transformers and langchain's openvinoembeddings. I'll also take a look. At the moment I'm trying to get gpt-oss to run on the server (int8 or int4,) but after a long period of loading, it fails with a "killed" message. I'll try 1.0.6 and see if I have better luck there.

[SearchSavior]

I've only used sentence transformers and langchain's openvinoembeddings. I'll also take a look. At the moment I'm trying to get gpt-oss to run on the server (int8 or int4,) but after a long period of loading, it fails with a "killed" message. I'll try 1.0.6 and see if I have better luck there.

Nice!

Check out this issue, and the linked PRs.

huggingface/optimum-intel#1275

Upstream is improving MoE support, should land in the next release, hopefully. Maintainers commented something similar in the gpt-oss pr. It seems they are using qwen moe to, I think, expand operation set coverage for new architectures, on top of enabling moe topologies in the device plugins which were not well optimized.

If you follow the build instructions in the issue I did see those performance gains. If you build openvino and point to the 1.0.5 env qwen3 moe works quite well.

[mwrothbe]

Well, no luck getting gpt-oss to load on 1.06 or 1.05. Not sure if I'm doing something silly. Is this a good place to air out details for possible pointers, or do you prefer discord (I realize model loading support isn't in scope for an "embeddings" thread)? If gpt-oss just not supported yet, that's fine and I'll move on. But I think there is a large chance I'm just doing it wrong. Also, if I read that optimum-intel PR correctly, it was closed with no merge to main. Hope it wasn't left on the floor as it sounds promising.

Details:
as per huggingface/optimum-intel#1428 I was able to optimize gpt-oss. I made three optimizations:
optimum-cli export openvino -m "openai/gpt-oss-20b" "models\gpt-oss-20b-int4" --weight-format int4
optimum-cli export openvino -m "openai/gpt-oss-20b" "models\gpt-oss-20b-int8" --weight-format int8
optimum-cli export openvino -m "openai/gpt-oss-20b" "models\gpt-oss-20b-int8a" --task text-generation-with-past --weight-format int8_asym

In 1.05 I attempted to load each one using "openarc load --model ~/openarc/models/gpt-oss-20b/gpt-oss-20b-intx/ --model-type TEXT --device AUTO --use-cache --dynamic-shapes" but after some time in loading, the server either crashes out completely or reports back "killed".

In 1.06 I used ovgenai engine as that seems to be the only engine with code so far: "openarc add --model-name gpt-oss --model-path ~/models/gpt-oss-20b-int8a/ --engine ovgenai --model-type llm --device AUTO". This results in a status of "loading" until it crashes out.

Any guidance\pointers would be appreciated. Thanks!

[SearchSavior]

Yeah, we're off emebddings now haha. Feel free to join discord anyway, or a fresh issue. Talking here is great too, lets people know the project is alive.

openarc add was supposed to be step 1, where you add to a config file then, load to server with openarc load --model-name after running openarc serve start in a different terminal. So maybe you are seeing expected behavior. Im working on documentation this weekend, thanks for checking it out!!

As for gpt-oss, it requires transformers 4.55, may have to upgrade. I haven't experimented with AutoTokenizers and Harmony format at all, so there are a few layers here to consider. Maybe try using one of the example scripts with less project level complexity for now to see how the model runs. I'll give it a whirl tonight and tag you in a fresh issue.

Imo don't use AUTO. The docs say it selects the best device but OpenArc implements openvino in an opinionated way now, so AUTO may create resource contention if you load more than one model.

Not sure if I'm going to implement optimum llm and vlm. Performance with genai has been much better... I don't see the need to maintain both. Instead we will use optimum engine for other tasks, like embeddings, with potential to extend to other engines in the future like AutoRound, triton kernels, whatever. Any thoughts?

openvino engine has only kokoro-82m implemented for now, have to tinker a bit more to get sse delivered correctly but it does work. That uses openvino directly, which I think is pretty cool.

[mwrothbe]

Thanks! Ya, I did figure out that the 1.06 paradigm is Add and then Load. I think I actually did: "openarc add --model-name gpt-oss --model-path ~/models/gpt-oss-20b-int8a/ --engine ovgenai --model-type llm --device AUTO" and then "load gpt-oss". It seemed to be doing the right thing. I could get a status of "loading", but it just fails to complete.

BTW, I initially had the Add model path wrong on my side, and the Add reported "successful" but the Status was always immediately "Failed" and I couldn't figure out why. In code saw that the status_message object has a property of error_message that doesn't get displayed in the status table. Once I hacked that to show the error message in the status report, I realized what was wrong. You might consider adding the error message to the status report when there is data.

Gotcha on not using AUTO. Seems to me I tried using "GPU" once, but maybe not "GPU.0". I might try that if you think it matters.

Yes, I am using Transformers 4.55.4. I just did a "uv pip install -U transformers" and that seemed to work. If you have an older version installed, it fails pretty quick highlighting a transformers issue.

Looking forward to hearing your findings if you get time to try it out!

Thanks!

[SearchSavior]

thanks for the review! Just got this behavior working

yeah use openarc tool dev-detect to see what devices are called in openvino, and what to use in --device. A special case would be HETERO device plugin.

[mwrothbe]

Thanks for implementing the error reporting! It's super useful! I found that when trying to load int8_asym optimized model, the 'oom-killer' was killing the process. I guess that means the model is too big for my A770. I then tried an int4 version (device:GPU.0), and now I'm getting:
"(openarc) (base) mike@UbuntuGPU1:~/OpenArc$ openarc load --mn gpt-oss
working...
❌ error: 500
Response: {"detail":"Failed to load model: Model loading failed: Exception from
src/inference/src/cpp/core.cpp:109:\nException from src/inference/src/dev/plugin.cpp:53:\nCheck 'false' failed at
src/plugins/intel_gpu/src/plugin/program_builder.cpp:163:\n[GPU] ProgramBuilder build failed!\nCheck 'shape_type ==
shape_types::dynamic_shape || node->selected_impl != nullptr' failed at
src/plugins/intel_gpu/src/graph/graph_optimizer/compile_graph.cpp:51:\n[GPU] Failed to select implementation
for\nname:convert:Convert_315928\ntype: reorder\noriginal_type: Convert Check '!kernels.empty()' failed at
src/plugins/intel_gpu/src/kernel_selector/kernel_selector.cpp:70:\n[GPU] Could not find a suitable kernel for
convert:Convert_315928 params raw string:
INT4_BFYX_v2880_p0_0_v32_p0_0_v90_p0_0_v32_p0_0;F16_BFYX_v2880_p0_0_v32_p0_0_v90_p0_0_v32_p0_0\n\n\n\n\n"}"

Any thoughts on what this could mean? My optimum cli was "optimum-cli export openvino -m "openai/gpt-oss-20b" "models\gpt-oss-20b-int4" --weight-format int4". Oddly enough adding as device: CPU crashes pretty consistently with no message aside from "killed" on the server. BTW, I'm using tmux to run the server on the same system as the cli. Not sure if there are issues doing it that way. It did seem to work for a DeepSeek model I was trying out before.

Thanks!

[SearchSavior]

@mwrothbe I just pushed better error handling. Now we display inference time failure stack trace and unload model from memory and registry. Needs some testing for LLM/VLM/Kokoro but it worked for whisper. LMK what you find with gpt-oss.

Couldnt see logs before because they were happening inside of an async loop. Applying that elsewhere was easy. That should help you debug.

gpt-oss error- this looks like maybe there are unimplemented operations gpt-oss requires. I think this is the important part is

[GPU] Could not find a suitable kernel for
convert:Convert_315928 params raw string:
INT4_BFYX_v2880_p0_0_v32_p0_0_v90_p0_0_v32_p0_0;F16_BFYX_v2880_p0_0_v32_p0_0_v90_p0_0_v32_p0_0\n\n\n\n\n"}"

Sonnet 4.5 reports;

Error Analysis
Root Cause: The GPU plugin cannot find a suitable kernel implementation for a specific conversion operation.
Specific Issues:

Failed Operation: Convert_315928 - a data type conversion node

Converting from: INT4 (4-bit integer)
Converting to: F16 (16-bit float/half precision)


Tensor Shape: BFYX format with dimensions: 2880 × 32 × 90 × 32

B = Batch: 2880
F = Features/Channels: 32
Y = Height: 90
X = Width: 32


Missing Kernel: The GPU kernel selector couldn't find an implementation that supports this specific INT4→F16 conversion with these dimensions.

AFAIK this is correct assessment. Had similar issues with Kokoro when I tried using GPU, have seen before similar in other openvino adventures lol. Maybe check the discussion in PRs leading up to merge, I think I remember seeing them discuss a choosing a default

[mwrothbe]

Well, back to the op, I got embeddings working on 1.0.6. However, I don't seem to be smart enough to figure out how to submit a PR.

[SearchSavior]

@mwrothbe Cool! I see you made a fork but dont see a commit into that fork. Maybe git push?

[mwrothbe]

alright, I think I figured it out. I tried to follow the structure as much as possible, but you may have to edit to your liking. This seems to be working with a qwen3 model using "optimum-cli export openvino --model "Qwen/Qwen3-Embedding-0.6B" --task "feature-extraction" "g:\models\qwen3_emb" --weight-format int8" and added as "openarc add --model-name qwen3_emb --model-path ~/Models/qwen3_emb/ --engine optimum --model-type emb --device HETERO:CPU". The POST is to /v1/embeddings with an example payload of {
"input": "Some text to embed",
"model": "qwen3_emb",
"dimensions": 1024,
"config":{
"padding":true,
"truncation":true,
"return_tensors":"pt"
}
}

[SearchSavior]

A very pleasant surprise! Looking forward to the review, and am glad you found this project helpful.

Note: hetero with single device would be incorrect, just --device CPU

[mwrothbe]

"Note: hetero with single device would be incorrect, just --device CPU"

ya, that was a typo I made in the CLI, but it still works. I guess it's smart enough to just treat it as CPU.

[SearchSavior]

interesting that it still worked lol. endless openvino mystery continues