Update dependency ai.djl.huggingface:tokenizers to v0.36.0

[renovate]

This PR contains the following updates:

Package	Type	Update	Change	OpenSSF
ai.djl.huggingface:tokenizers (source)	compile	minor	0.33.0 → 0.36.0

---

Release Notes

deepjavalibrary/djl (ai.djl.huggingface:tokenizers)

v0.36.0: DJL v0.36.0 Release

Compare Source

What's Changed

• Increase build version to 0.36.0 by @​access2rohit in #​3817
• Update the Documentation (via DocString) to indicate the risks of using Utils.openUrl() with untrusted inputs by @​access2rohit in #​3819
• NDManager.create(Number) unsupported type error message improvement by @​fracpete in #​3818

New Contributors

• @​access2rohit made their first contribution in #​3816
• @​fracpete made their first contribution in #​3818

Full Changelog: deepjavalibrary/djl@v0.35.1...v0.36.0

v0.35.1: DJL v0.35.1 Release

Compare Source

What's Changed

• fix: upgrade jQuery from 2.1.1 to 3.7.1 to address security vulnerabilities by @​Lokiiiiii in #​3812
• Fix script injection vulnerabilities in GitHub Actions workflows by @​Lokiiiiii in #​3813
• Fix security vulnerabilities in QuestionAnsweringTranslator (CVSS 8.2) by @​Lokiiiiii in #​3814

New Contributors

• @​Lokiiiiii made their first contribution in #​3812

Full Changelog: deepjavalibrary/djl@v0.35.0...v0.35.1

v0.35.0: DJL v0.35.0 Release

Compare Source

What's Changed

• Engine Updates
  • XGBoost 3.0.4 #​3788

Enhancements

• Increase build version to 0.35.0 by @​xyang16 in #​3766
• Add default header for openai by @​bryanktliu in #​3772
• Update gradle to 9.0.0 by @​xyang16 in #​3777
• [XGBoost] Support json model for XGBoost by @​xyang16 in #​3786
• [api] Accept json input/output for tensor input by @​xyang16 in #​3787
• Upgrade xgboost to 3.0.4 by @​xyang16 in #​3788
• [pytorch] Add tuple of dict support for IValue by @​xyang16 in #​3797
• [XGBoost] Add back gpu support for xgboost by @​xyang16 in #​3796
• Update gradle build script to support configuration cache by @​xyang16 in #​3782
• Added percentile implementation for Pytorch engine by @​tipame in #​3800
• Implemented CSV translator that handles CSV input/output support for XGBoost by @​smouaa in #​3801
• Correct topK documentation by @​alanocallaghan in #​3808
• [api] Refactor csvTranslator by @​xyang16 in #​3805

Bug Fixes

• Fix pytorch native 2.5.1 aarch64 build by @​xyang16 in #​3767
• [fix] Fixes load tokenizer gpu shared library issue by @​xyang16 in #​3773
• Fixes #​3774 for YoloSegmentationTranslator by @​bryanktliu in #​3775
• Fixes issue #​3761 by @​bryanktliu in #​3776
• Fix BaseNDManager.debugDump by @​petebankhead in #​3780
• Fix preference of tokenizer_config.json and remove doLowerCase from TokenizerConfig by @​Soha-Agarwal in #​3785
• Fixes: #​3795, adds aggregation_strategy to token classification task by @​bryanktliu in #​3798
• Fix pytorch native build for windows by @​xyang16 in #​3802

CI/CD

• Build pytorch jni cpu precxx by @​xyang16 in #​3770
• Remove pytorch 2.1.2 and 2.3.1 publish in nightly by @​xyang16 in #​3771
• Fix sentencepiece native workflow by @​xyang16 in #​3768
• Fix fasttext native workflow by @​xyang16 in #​3769
• Fix continuous build by @​xyang16 in #​3804

New Contributors

• @​smouaa made their first contribution in #​3801
• @​alanocallaghan made their first contribution in #​3808

Full Changelog: deepjavalibrary/djl@v0.34.0...v0.35.0

v0.34.0: DJL v0.34.0 Release

Compare Source

Key Changes

• Engine Updates
  • PyTorch 2.7.1 #​3733
  • Remove TensorRT engine #​3765

Enhancements

• [api] Adds FUSE based repository support by @​raymondkhliu in #​3695
• Treat model server as a remote model by @​xyang16 in #​3704
• Align ZeroShotClassificationTranslator with Hugging Face behavior by @​raphaeldelio in #​3712
• Remove usage of UUID.randomUUID() for better performance by @​aakashb-kayzen in #​3719
• Replace UUID with NDManager.nextUid() by @​xyang16 in #​3720
• Add NDArray diagonal by @​dev-jonghoonpark in #​3724
• [api] Implements remote REST API call as a model by @​bryanktliu in #​3725
• Adds jsonlines streaming support for RpcEngine by @​bryanktliu in #​3727
• Adds genai extension by @​bryanktliu in #​3726
• [genai] tool calls to ChatInput by @​bryanktliu in #​3728
• Add utility class for genai function call by @​bryanktliu in #​3729
• Improve function call for genai extension by @​bryanktliu in #​3731
• Support convert ChatInput to GeminiInput by @​bryanktliu in #​3739
• Add partial support for HuggingFace Tokenizer to use parameters from tokenizer_config.json by @​Soha-Agarwal in #​3738
• Add Anthropic support from genai by @​bryanktliu in #​3759

Bug Fixes

• Correct CaptchaDataset options count to prevent torch.gather IndexError by @​xinhuagu in #​3694
• Fix dimension check when reshaping offsets in SeqBatcher by @​xinhuagu in #​3702
• Fix the incorrect function name and minor typo in doc by @​xinhuagu in #​3703
• Fix incorrect return value in getImageHeight method by @​xinhuagu in #​3706
• [djl_converter]: Fixes djl_converter bug by @​raymondkhliu in #​3705
• Fix the out of bounds exception when using an init image by @​luke-zhou in #​3717
• Fix earlystopping metric issue #​3722 by @​SamBSalgado in #​3736
• [tokenizers] Fix tokenizer cpu and cuda build by @​xyang16 in #​3756
• [examples] Update tfhub urls by @​xyang16 in #​3752
• Update tfhub repository url by @​xyang16 in #​3755

Documentation

• [docs] Update model zoo document by @​xyang16 in #​3690

CI/CD

• increase build version to 0.34.0 by @​siddvenk in #​3689
• Uses new recommended way to build pip wheel by @​raymondkhliu in #​3709
• Bump protobuf from 3.20.2 to 4.25.8 in /extensions/tokenizers/src/main/python by @​dependabot[bot] in #​3716
• [android] Upgrade DJL version to 0.34.0 by @​xyang16 in #​3721
• [ci] simplify nightly publish workflow by @​xyang16 in #​3732
• [ci] Update maven central publish url and credential by @​xyang16 in #​3748
• [djl-converter] Update license in pyproject.toml by @​xyang16 in #​3749
• [pytorch] Updates PyTorch to 2.7.1 by @​xyang16 in #​3733
• [ci] Remove serving-publish workflow by @​xyang16 in #​3750
• Update oss sonatype snapshot repository url by @​xyang16 in #​3754
• Upgrade dependency versions by @​xyang16 in #​3762
• [android] Upgrade pytorch version to 2.7.1 by @​xyang16 in #​3763
• [ci] Add post-publish task by @​xyang16 in #​3764
• Increase DJL version to 0.34.0 by @​Soha-Agarwal in #​3758

New Contributors

• @​raymondkhliu made their first contribution in #​3695
• @​raphaeldelio made their first contribution in #​3712
• @​luke-zhou made their first contribution in #​3717
• @​aakashb-kayzen made their first contribution in #​3719
• @​dev-jonghoonpark made their first contribution in #​3724
• @​Soha-Agarwal made their first contribution in #​3738
• @​SamBSalgado made their first contribution in #​3736

Full Changelog: deepjavalibrary/djl@v0.33.0...v0.34.0

---

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[bjorncs]

Let's upgrade this before adding back #36071.

[hmusum]

Fails to build with:

[ERROR] HuggingFaceTokenizerTest.bert_tokenizer:35 » NoClassDefFound Could not initialize class ai.djl.huggingface.tokenizers.jni.LibUtils

2026-03-11 09:37:31 CEST | [ERROR] HuggingFaceTokenizerTest.pads_to_max_length:109 » NoClassDefFound Could not initialize class ai.djl.huggingface.tokenizers.jni.LibUtils
2026-03-11 09:37:31 CEST | [ERROR] HuggingFaceTokenizerTest.provides_model_info:121 » NoClassDefFound Could not initialize class ai.djl.huggingface.tokenizers.jni.LibUtils
2026-03-11 09:37:31 CEST | [ERROR] HuggingFaceTokenizerTest.tokenizes_using_paraphrase_multilingual_mpnet_base_v2:52 » UnsatisfiedLink /root/.djl.ai/tokenizers/0.21.0-0.36.0-cpu-linux-aarch64/libtokenizers.so: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /root/.djl.ai/tokenizers/0.21.0-0.36.0-cpu-linux-aarch64/libtokenizers.so)
2026-03-11 09:37:31 CEST | [ERROR] HuggingFaceTokenizerTest.truncates_to_max_length:91 » NoClassDefFound Could not initialize class ai.djl.huggingface.tokenizers.jni.LibUtils

[bjorncs]

The build fails on AlmaLinux 8 because the DJL tokenizers v0.36.0 native library (libtokenizers.so) requires glibc 2.29, but AlmaLinux 8 ships glibc 2.28.

The only failing module is linguistics-components — all 5 test failures are in HuggingFaceTokenizerTest:

Test	Error
tokenizes_using_paraphrase_multilingual_mpnet_base_v2	UnsatisfiedLinkError — root cause
bert_tokenizer	NoClassDefFoundError — cascading
pads_to_max_length	NoClassDefFoundError — cascading
provides_model_info	NoClassDefFoundError — cascading
truncates_to_max_length	NoClassDefFoundError — cascading

[bjorncs]

After discussing with @arnej27959 and @toregge: the native so library requires at least glibc 2.34, which will be available once we have upgraded to AlmaLinux 9.