Respect tokenizer.json padding settings in HuggingFaceEmbedder

[s-smits]

Fix #35600

Summary

HuggingFaceEmbedder hardcoded .setPadding(false), ignoring padding configuration from tokenizer.json. Models like siglip2-base-patch16-384 that require fixed-length padding produced incorrect embeddings.

Replace hardcoded false with info.padding() != DO_NOT_PAD, which comes from the tokenizer's own metadata via ModelInfo.

Backwards compatible: existing tokenizers without padding config have padding == DO_NOT_PAD, producing the same setPadding(false) as before.

Changes

• HuggingFaceEmbedder.java: .setPadding(false) → .setPadding(info.padding() != DO_NOT_PAD)

I've also included a test (testPadding() in HuggingFaceEmbedderTest) and a test fixture (tokenizer_with_padding.json) to verify the fix. I can adjust or remove these if you prefer a different testing approach.

Test plan

• New testPadding() verifies padded tokenizer produces 10 tokens, unpadded produces 6
• All 20 existing embedder tests pass (HuggingFace, ColBert, Splade)

---

Claude Code has been used to assist with this PR.

[s-smits]

Testing note: All 8 existing tests in HuggingFaceEmbedderTest pass unchanged, real_tokenizer.json has padding: null (DO_NOT_PAD), so the fix produces identical behavior to the old hardcode.

A dedicated test could look like:

@Test
void testPaddingRespectedFromTokenizer() {
    var tokenizerPath = Paths.get("src/test/models/onnx/transformer/real_tokenizer.json");
    var info = HuggingFaceTokenizer.getModelInfo(tokenizerPath);

    assertEquals(ModelInfo.PaddingStrategy.DO_NOT_PAD, info.padding());
}

[glebashnik]

Looks good!