PaddleOCR-VL-1.5微调出错

[Moyhub]

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

[rank31]: Traceback (most recent call last):
[rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/cli/sft.py", line 20, in
[rank31]: sft_main()
[rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/pipelines/train/sft.py", line 346, in sft_main
[rank31]: return SwiftSft(args).main()
[rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/pipelines/train/sft.py", line 30, in init
[rank31]: self._prepare_model_tokenizer()
[rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/ray/base.py", line 169, in wrapper
[rank31]: return func(self, *args, **kwargs)
[rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/pipelines/train/sft.py", line 52, in _prepare_model_tokenizer
[rank31]: self.model, self.processor = args.get_model_processor(**kwargs)
[rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/arguments/base_args/base_args.py", line 328, in get_model_processor
[rank31]: return get_model_processor(**res)
[rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/model/register.py", line 607, in get_model_processor
[rank31]: return loader.load()
[rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/model/register.py", line 456, in load
[rank31]: model, processor = self._get_model_processor(model_dir, config)
[rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/model/register.py", line 447, in _get_model_processor
[rank31]: model = self.get_model(model_dir, config, processor, self.model_kwargs.copy())
[rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/model/models/baidu.py", line 98, in get_model
[rank31]: return super().get_model(model_dir, *args, **kwargs)
[rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/model/register.py", line 306, in get_model
[rank31]: model = auto_model_cls.from_pretrained(model_dir, config=config, trust_remote_code=True, **model_kwargs)
[rank31]: File "/usr/local/python3.10/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 607, in from_pretrained
[rank31]: raise ValueError(
[rank31]: ValueError: Unrecognized configuration class <class 'transformers_modules.PaddleOCR_hyphen_VL_hyphen_1_5.configuration_paddleocr_vl.PaddleOCRVLConfig'> for this kind of AutoModel: AutoModelForImageTextToText.
[rank31]: Model type should be one of AriaConfig, AyaVisionConfig, BlipConfig, Blip2Config, ChameleonConfig, Cohere2VisionConfig, DeepseekVLConfig, DeepseekVLHybridConfig, Emu3Config, EvollaConfig, Florence2Config, FuyuConfig, Gemma3Config, Gemma3nConfig, GitConfig, Glm4vConfig, Glm4vMoeConfig, GotOcr2Config, IdeficsConfig, Idefics2Config, Idefics3Config, InstructBlipConfig, InternVLConfig, JanusConfig, Kosmos2Config, Kosmos2_5Config, Lfm2VlConfig, Llama4Config, LlavaConfig, LlavaNextConfig, LlavaNextVideoConfig, LlavaOnevisionConfig, Mistral3Config, MllamaConfig, Ovis2Config, PaliGemmaConfig, PerceptionLMConfig, Pix2StructConfig, PixtralVisionConfig, Qwen2_5_VLConfig, Qwen2VLConfig, Qwen3VLConfig, Qwen3VLMoeConfig, ShieldGemma2Config, SmolVLMConfig, UdopConfig, VipLlavaConfig, VisionEncoderDecoderConfig.

How to Reproduce / 如何复现

运行环境：NPU

脚本：
MIN_PIXELS=$MIN_PIXELS
MAX_PIXELS=$MAX_PIXELS
torchrun
--nproc_per_node $NPUS_PER_NODE
--nnodes $NNODES
--node_rank $NODE_RANK
--master_addr $MASTER_ADDR
--master_port $MASTER_PORT
swift/cli/sft.py
--model_type paddle_ocr_1_5
--model $BASE_MODEL_DIR
--dataset $DATA_PATH
$resume_param
--train_type full
--torch_dtype float16
--num_train_epochs 6
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--learning_rate 2e-5
--gradient_accumulation_steps 4
--save_steps 1000
--eval_strategy no
--save_total_limit 2
--logging_steps 10
--max_length 5600
--output_dir /home/kas/pretraining/saved_models/layout/${MODEL_ID}
--warmup_ratio 0.05
--lazy_tokenize true
--ddp_backend hccl
--freeze_aligner false
--lr_scheduler_type=cosine
--freeze_vit false
--deepspeed zero2
--dataloader_num_workers 8 2>&1 | tee -a ${LOG_DIR}/rank${NODE_RANK}_train.log

Additional Information / 补充信息

No response

[slin000111]

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

[rank31]: Traceback (most recent call last): [rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/cli/sft.py", line 20, in [rank31]: sft_main() [rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/pipelines/train/sft.py", line 346, in sft_main [rank31]: return SwiftSft(args).main() [rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/pipelines/train/sft.py", line 30, in init [rank31]: self._prepare_model_tokenizer() [rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/ray/base.py", line 169, in wrapper [rank31]: return func(self, *args, **kwargs) [rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/pipelines/train/sft.py", line 52, in _prepare_model_tokenizer [rank31]: self.model, self.processor = args.get_model_processor(**kwargs) [rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/arguments/base_args/base_args.py", line 328, in get_model_processor [rank31]: return get_model_processor(**res) [rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/model/register.py", line 607, in get_model_processor [rank31]: return loader.load() [rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/model/register.py", line 456, in load [rank31]: model, processor = self._get_model_processor(model_dir, config) [rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/model/register.py", line 447, in _get_model_processor [rank31]: model = self.get_model(model_dir, config, processor, self.model_kwargs.copy()) [rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/model/models/baidu.py", line 98, in get_model [rank31]: return super().get_model(model_dir, *args, **kwargs) [rank31]: File "/home/kas/pretraining/ms-swift-3.12.4/swift/model/register.py", line 306, in get_model [rank31]: model = auto_model_cls.from_pretrained(model_dir, config=config, trust_remote_code=True, **model_kwargs) [rank31]: File "/usr/local/python3.10/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 607, in from_pretrained [rank31]: raise ValueError( [rank31]: ValueError: Unrecognized configuration class <class 'transformers_modules.PaddleOCR_hyphen_VL_hyphen_1_5.configuration_paddleocr_vl.PaddleOCRVLConfig'> for this kind of AutoModel: AutoModelForImageTextToText. [rank31]: Model type should be one of AriaConfig, AyaVisionConfig, BlipConfig, Blip2Config, ChameleonConfig, Cohere2VisionConfig, DeepseekVLConfig, DeepseekVLHybridConfig, Emu3Config, EvollaConfig, Florence2Config, FuyuConfig, Gemma3Config, Gemma3nConfig, GitConfig, Glm4vConfig, Glm4vMoeConfig, GotOcr2Config, IdeficsConfig, Idefics2Config, Idefics3Config, InstructBlipConfig, InternVLConfig, JanusConfig, Kosmos2Config, Kosmos2_5Config, Lfm2VlConfig, Llama4Config, LlavaConfig, LlavaNextConfig, LlavaNextVideoConfig, LlavaOnevisionConfig, Mistral3Config, MllamaConfig, Ovis2Config, PaliGemmaConfig, PerceptionLMConfig, Pix2StructConfig, PixtralVisionConfig, Qwen2_5_VLConfig, Qwen2VLConfig, Qwen3VLConfig, Qwen3VLMoeConfig, ShieldGemma2Config, SmolVLMConfig, UdopConfig, VipLlavaConfig, VisionEncoderDecoderConfig.

How to Reproduce / 如何复现

运行环境：NPU

脚本： MIN_PIXELS=$MIN_PIXELS MAX_PIXELS=$MAX_PIXELS torchrun --nproc_per_node $NPUS_PER_NODE --nnodes $NNODES --node_rank $NODE_RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT swift/cli/sft.py --model_type paddle_ocr_1_5 --model $BASE_MODEL_DIR --dataset $DATA_PATH $resume_param --train_type full --torch_dtype float16 --num_train_epochs 6 --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --learning_rate 2e-5 --gradient_accumulation_steps 4 --save_steps 1000 --eval_strategy no --save_total_limit 2 --logging_steps 10 --max_length 5600 --output_dir /home/kas/pretraining/saved_models/layout/${MODEL_ID} --warmup_ratio 0.05 --lazy_tokenize true --ddp_backend hccl --freeze_aligner false --lr_scheduler_type=cosine --freeze_vit false --deepspeed zero2 --dataloader_num_workers 8 2>&1 | tee -a ${LOG_DIR}/rank${NODE_RANK}_train.log

Additional Information / 补充信息

No response

transformers升级一下

[Moyhub]

transformers 4.57.0
升级到哪个版本呢？这是我的transformer版本

[slin000111]

transformers 4.57.0 升级到哪个版本呢？这是我的transformer版本

transformers>=5.0.0

[dhhcj1]

transformers 4.57.0 升级到哪个版本呢？这是我的transformer版本

transformers>=5.0.0

我目前也想训练paddleocr vl1.5， 请问ms-swfit 版本必须要升级到4.0吗

[slin000111]

transformers 4.57.0 升级到哪个版本呢？这是我的transformer版本

transformers>=5.0.0

我目前也想训练paddleocr vl1.5， 请问ms-swfit 版本必须要升级到4.0吗

应该还在ms-swift main分支

[dhhcj1]

transformers 4.57.0 升级到哪个版本呢？这是我的transformer版本

transformers>=5.0.0

我目前也想训练paddleocr vl1.5， 请问ms-swfit 版本必须要升级到4.0吗

应该还在ms-swift main分支

那麻烦在请教一下是具体在分支的哪个版本呢？

[slin000111]

transformers 4.57.0 升级到哪个版本呢？这是我的transformer版本

transformers>=5.0.0

我目前也想训练paddleocr vl1.5， 请问ms-swfit 版本必须要升级到4.0吗

应该还在ms-swift main分支

那麻烦在请教一下是具体在分支的哪个版本呢？

最新代码安装ms-swift

git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e .

[dhhcj1]

感谢大佬

…

---原始邮件--- 发件人: ***@***.***&gt; 发送时间: 2026年2月6日(周五) 上午10:39 收件人: ***@***.***&gt;; 抄送: ***@***.******@***.***&gt;; 主题: Re: [modelscope/ms-swift] PaddleOCR-VL-1.5微调出错 (Issue #7997) slin000111 left a comment (modelscope/ms-swift#7997) transformers 4.57.0 升级到哪个版本呢？这是我的transformer版本 transformers&gt;=5.0.0 我目前也想训练paddleocr vl1.5， 请问ms-swfit 版本必须要升级到4.0吗 应该还在ms-swift main分支 那麻烦在请教一下是具体在分支的哪个版本呢？ 最新代码安装ms-swift git clone https://github.com/modelscope/ms-swift.git cd ms-swift pip install -e . — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***&gt;

[Moyhub]

@slin000111 升级到5.0.0后
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/transformers/tokenization_mistral_common.py", line 43, in
[rank0]: from mistral_common.tokens.tokenizers.base import SpecialTokenPolicy, SpecialTokens
[rank0]: ImportError: cannot import name 'SpecialTokenPolicy' from 'mistral_common.tokens.tokenizers.base' (/usr/local/python3.10/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/base.py)
环境错误。

另外就是pip install -e . 中的requestment transformer版本没有更新，transformers>=4.33,<4.58

[slin000111]

@slin000111 升级到5.0.0后 [rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/transformers/tokenization_mistral_common.py", line 43, in [rank0]: from mistral_common.tokens.tokenizers.base import SpecialTokenPolicy, SpecialTokens [rank0]: ImportError: cannot import name 'SpecialTokenPolicy' from 'mistral_common.tokens.tokenizers.base' (/usr/local/python3.10/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/base.py) 环境错误。

另外就是pip install -e . 中的requestment transformer版本没有更新，transformers>=4.33,<4.58

安装ms-swift后再升级transformers就行。~报错看起来是mistral相关的。可以先运行一下模型卡片上的例子，看看环境正常的不。

from PIL import Image
import torch
from modelscope import AutoProcessor, AutoModelForImageTextToText

# ---- Settings ----
model_path = "PaddlePaddle/PaddleOCR-VL-1.5"
image_path = "test.png"
task = "ocr" # Options: 'ocr' | 'table' | 'chart' | 'formula' | 'spotting' | 'seal'
# ------------------

# ---- Image Preprocessing For Spotting ----
image = Image.open(image_path).convert("RGB")
orig_w, orig_h = image.size
spotting_upscale_threshold = 1500

if task == "spotting" and orig_w < spotting_upscale_threshold and orig_h < spotting_upscale_threshold:
    process_w, process_h = orig_w * 2, orig_h * 2
    try:
        resample_filter = Image.Resampling.LANCZOS
    except AttributeError:
        resample_filter = Image.LANCZOS
    image = image.resize((process_w, process_h), resample_filter)

# Set max_pixels: use 1605632 for spotting, otherwise use default ~1M pixels
max_pixels = 2048 * 28 * 28 if task == "spotting" else 1280 * 28 * 28
# ---------------------------

# -------- Inference --------
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
PROMPTS = {
    "ocr": "OCR:",
    "table": "Table Recognition:",
    "formula": "Formula Recognition:",
    "chart": "Chart Recognition:",
    "spotting": "Spotting:",
    "seal": "Seal Recognition:",
}

model = AutoModelForImageTextToText.from_pretrained(model_path, torch_dtype=torch.bfloat16).to(DEVICE).eval()
processor = AutoProcessor.from_pretrained(model_path)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": PROMPTS[task]},
        ]
    }
]
inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    images_kwargs={"size": {"shortest_edge": processor.image_processor.min_pixels, "longest_edge": max_pixels}},
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
result = processor.decode(outputs[0][inputs["input_ids"].shape[-1]:-1])
print(result)
# ---------------------------

[Moyhub]

嗯嗯是的，我模型卡片上的代码，只保留模型加载的部分，也会报和上述一样的问题。所以mistral_common安装包版本是多少呢？我现在是1.5.3、

[slin000111]

嗯嗯是的，我模型卡片上的代码，只保留模型加载的部分，也会报和上述一样的问题。所以mistral_common安装包版本是多少呢？我现在是1.5.3、

试一下这个版本

mistral_common                           1.8.6

[slin000111]

嗯嗯是的，我模型卡片上的代码，只保留模型加载的部分，也会报和上述一样的问题。所以mistral_common安装包版本是多少呢？我现在是1.5.3、

一般可以参考mistral最新模型，模型卡片上有依赖。