Support Paraformer-zh in optimum-intel

[padatta]

Summary

This PR adds support for exporting Alibaba's Paraformer ASR model (funasr/paraformer-zh) to OpenVINO IR format with comprehensive INT8 quantization capabilities and full inference support.

Key Features

• Full Paraformer model export to OpenVINO IR format (FP16)
• INT8 full quantization with AISHELL-1 dataset calibration
• GPU-compatible indexed operations using torch.where and index_fill_
• Per-tensor activation quantization for optimal inference performance
• Auto-detection of Paraformer models via characteristic files (am.mvn, config.yaml, tokens.json)
• OpenVINO inference model (OVParaformerForSpeechSeq2Seq) following the SpeechT5TTS pattern

Files Changed

Export Implementation

File	Change
optimum/exporters/openvino/modeling_paraformer.py	NEW - Complete export implementation (2118 lines)
optimum/exporters/openvino/__main__.py	INT8 quantization pipeline with NNCF integration
optimum/commands/export/openvino.py	CLI command updates for paraformer library support
optimum/intel/openvino/utils.py	Added aishell-1 to predefined speech-to-text datasets
optimum/intel/utils/modeling_utils.py	Paraformer model auto-detection

Inference Implementation

File	Change
optimum/intel/openvino/modeling_speech2text.py	NEW - Full inference implementation (847 lines)
optimum/intel/__init__.py	Export OVParaformerForSpeechSeq2Seq
optimum/intel/openvino/__init__.py	Import from modeling_speech2text
optimum/intel/pipelines/accelerator_utils.py	Add to ASR pipeline support
optimum/intel/utils/dummy_openvino_objects.py	Add dummy object

Usage

Export Models

# FP16 Export
optimum-cli export openvino --trust-remote-code --model funasr/paraformer-zh ov_paraformer_fp16

# INT8 Full Quantization
optimum-cli export openvino --trust-remote-code --model funasr/paraformer-zh \
    --quant-mode int8 --dataset aishell-1 ov_paraformer_int8

Inference with OVParaformerForSpeechSeq2Seq

from optimum.intel.openvino import OVParaformerForSpeechSeq2Seq
import torch

Load model

model = OVParaformerForSpeechSeq2Seq.from_pretrained(
"ov_paraformer_int8/ov_models",
device="GPU"
)

Prepare inputs

speech = torch.randn(1, 100, 560) # [batch, time, features]
speech_lengths = torch.tensor([100], dtype=torch.int32)

Run inference

output = model(speech, speech_lengths)

Get results

token_ids = output.token_ids # Decoded token IDs
token_num = output.token_num # Number of valid tokens
logits = output.logits # Raw logits

[rkazants]

Please add tests,
openvino.py, main.py should not be changed. Please check what files are changed when adding new models support in other PRs.
Thanks

[Copilot]

Pull request overview

This PR introduces Paraformer (funasr/paraformer-zh) support in optimum-intel’s OpenVINO workflow, covering model auto-detection, export to OpenVINO IR, optional INT8 quantization, and a new OpenVINO inference wrapper class.

Changes:

• Add Paraformer auto-detection and pipeline registration for ASR.
• Implement Paraformer OpenVINO export path (including INT8 quantization flow).
• Add a new OpenVINO inference implementation OVParaformerForSpeechSeq2Seq.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
optimum/intel/utils/modeling_utils.py	Detect Paraformer models via presence of am.mvn.
optimum/intel/utils/dummy_openvino_objects.py	Add dummy OVParaformerForSpeechSeq2Seq for missing OpenVINO backend.
optimum/intel/pipelines/accelerator_utils.py	Register Paraformer class under ASR task mapping.
optimum/intel/openvino/utils.py	Add aishell-1 dataset marker and adjust predefined dataset metadata.
optimum/intel/openvino/modeling_speech2text.py	New Paraformer OpenVINO inference wrapper and component classes.
optimum/intel/openvino/__init__.py	Export OVParaformerForSpeechSeq2Seq from OpenVINO package.
optimum/intel/__init__.py	Export OVParaformerForSpeechSeq2Seq from top-level package.
optimum/exporters/openvino/modeling_paraformer.py	New Paraformer Torch/TorchScript export implementation used by OpenVINO conversion.
optimum/exporters/openvino/__main__.py	Add Paraformer-specific export + quantization handling into main_export and task inference.
optimum/commands/export/openvino.py	Skip _main_quantize for Paraformer since quantization is handled during export.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Paraformer INT8 quantization imports librosa unconditionally inside the quantization branch. librosa is not a declared runtime dependency, so --quant-mode int8 --dataset aishell-1 will crash with ModuleNotFoundError in standard installs. Either switch to the existing datasets-based audio loading used elsewhere in the repo, or wrap the import in a try/except and raise a targeted error telling users which extra to install.

Suggested change

	import librosa
	try:
	import librosa
	except ModuleNotFoundError as exc:
	if exc.name == "librosa":
	raise RuntimeError(
	"INT8 quantization for Paraformer requires the optional 'librosa' dependency for "
	"audio feature extraction. Please install it with `pip install librosa` or install "
	"the appropriate audio extra for this package."
	) from exc
	raise

[Copilot]

Both StreamSinusoidalPositionEncoder and SinusoidalPositionEncoder define __int__ instead of __init__, so their constructors will never run. If these classes are expected to initialize any state (e.g., dropout), it will silently be skipped. Rename __int__ to __init__ (or remove the method entirely if no initialization is needed).

Suggested change

	def __int__(self, d_model=80, dropout_rate=0.1):
	pass
	# Copied from https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/embedding.py#L383 (Apache 2.0)
	class SinusoidalPositionEncoder(torch.nn.Module):
	""" """
	def __int__(self, d_model=80, dropout_rate=0.1):
	pass
	def __init__(self, d_model=80, dropout_rate=0.1):
	super().__init__()
	# Copied from https://github.com/modelscope/FunASR/blob/main/funasr/models/transformer/embedding.py#L383 (Apache 2.0)
	class SinusoidalPositionEncoder(torch.nn.Module):
	""" """
	def __init__(self, d_model=80, dropout_rate=0.1):
	super().__init__()

[Copilot]

lora.Linear / lora.MergedLinear is referenced when lora_list is not None, but lora is never imported/defined in this module. This will raise NameError for configs that enable LoRA. Add the missing import (or gate the LoRA branch behind an availability check with a clear error).

[Copilot]

OVParaformerPredictor.forward() builds an attention mask but feeds it into the input named "encoder_out_lens", while also hard-coding input names ("encoder_out", "encoder_out_lens"). This is very likely to mismatch the exported predictor IR inputs (which typically follow the TorchScript arg names, e.g. hidden/mask) and will break component-based inference. Use the discovered self.input_names (or positional indices) to map encoder_out and the computed mask to the actual OpenVINO input names.

Suggested change

	inputs = {
	"encoder_out": self._prepare_input(encoder_out),
	"encoder_out_lens": self._prepare_input(mask),
	}
	# Map encoder_out and mask to actual OV input names using discovered input_names
	inputs = {}
	if len(self.input_names) > 0:
	inputs[self.input_names[0]] = self._prepare_input(encoder_out)
	if len(self.input_names) > 1:
	inputs[self.input_names[1]] = self._prepare_input(mask)

[Copilot]

PREDEFINED_SD_DATASETS entries no longer include an id, but the quantization pipeline expects dataset_metadata["id"] (see optimum/intel/openvino/quantization.py). This will raise a KeyError when quantizing diffusion pipelines. Add the missing id back for each predefined SD dataset (or update the quantization code to handle missing ids consistently).

[Copilot]

The raised ValueError message mentions the optimum repo and "ONNX export", but this codepath is in the OpenVINO exporter and other messages in this file point users to huggingface/optimum-intel and OpenVINO export. Update this message to consistently reference the correct repository and exporter to avoid confusing users.

Suggested change

f"Asked to export a {model_type} model for the task {task}{autodetected_message}, but the Optimum OpenVINO exporter only supports the tasks {', '.join(model_tasks.keys())} for {model_type}. Please use a supported task. Please open an issue at https://github.com/huggingface/optimum/issues if you would like the task {task} to be supported in the ONNX export for {model_type}."

f"Asked to export a {model_type} model for the task {task}{autodetected_message}, but the Optimum OpenVINO exporter only supports the tasks {', '.join(model_tasks.keys())} for {model_type}. Please use a supported task. Please open an issue at https://github.com/huggingface/optimum-intel/issues if you would like the task {task} to be supported in the OpenVINO export for {model_type}."

[Copilot]

This module hard-depends on omegaconf, but omegaconf is not listed in install_requires/extras. As-is, Paraformer export will fail at import time with ModuleNotFoundError. Either add omegaconf to the appropriate extra (and ensure the exporter only imports this module when that extra is installed), or remove the dependency by using standard YAML/JSON parsing.

Suggested change

	from omegaconf import OmegaConf, DictConfig, ListConfig
	try:
	from omegaconf import OmegaConf, DictConfig, ListConfig
	_OMEGACONF_AVAILABLE = True
	except ImportError: # pragma: no cover - optional dependency
	OmegaConf = None # type: ignore[assignment]
	DictConfig = None # type: ignore[assignment]
	ListConfig = None # type: ignore[assignment]
	_OMEGACONF_AVAILABLE = False

[Copilot]

There are unused imports (hf_hub_download, GenerationConfig, and Model), which will trigger Ruff F401 failures and increase import time. Remove them or use them as intended (e.g., implement Hub downloading if planned).

Suggested change

	from openvino import CompiledModel, Core, Model
	import torch
	from huggingface_hub import hf_hub_download
	from huggingface_hub.constants import HUGGINGFACE_HUB_CACHE
	from transformers import AutoConfig, PretrainedConfig, GenerationConfig
	from openvino import CompiledModel, Core
	import torch
	from huggingface_hub.constants import HUGGINGFACE_HUB_CACHE
	from transformers import AutoConfig, PretrainedConfig

[Copilot]

OV_TASKS_MAPPING["automatic-speech-recognition"] now includes OVParaformerForSpeechSeq2Seq, but get_openvino_model_class() only ever returns index 0 (CTC) or index 1 (seq2seq). As a result, Paraformer will never be auto-selected by pipelines for ASR. Either update the selection logic to detect Paraformer models (e.g., via library_name/characteristic files) and return the Paraformer class, or remove it from the mapping to avoid a misleading entry.

[padatta]

Please add tests, openvino.py, main.py should not be changed. Please check what files are changed when adding new models support in other PRs. Thanks

Please add tests, openvino.py, main.py should not be changed. Please check what files are changed when adding new models support in other PRs. Thanks

I’ve added tests for the Paraformer model and refactored the export logic. There are no changes to main.py or openvino.py, and the implementation follows the same conventions used for the other models.
Please review the changes and let me know if any additional updates are needed.