Add XLMRoBERTaModel in paddlenlp by jie-z-0607 · Pull Request #9720 · PaddlePaddle/PaddleNLP

jie-z-0607 · 2024-12-31T03:41:54Z

PR types

New features

PR changes

Models

Description

在PaddleNLP中增加对于XLM-RoBERTa系列模型的支持，已支持相关预训练模型如下：

model name	model type
BAAI/bge-m3	XLMRobertaModel
BAAI/bge-reranker-v2-m3	XLMRobertaForSequenceClassification
BAAI/bge-reranker-large	XLMRobertaForSequenceClassification
BAAI/bge-reranker-base	XLMRobertaForSequenceClassification
BAAI/bge-m3-unsupervised	XLMRobertaModel

JunnYu · 2024-12-31T03:53:12Z

+    Examples:
+
+    ```python
+    >>> from ppdiffusers.transformers import XLMRobertaConfig, XLMRobertaModel


改一下文档

JunnYu · 2024-12-31T03:54:00Z

+        classifier_dropout=None,
+        **kwargs,
+    ):
+        kwargs["return_dict"] = kwargs.pop("return_dict", True)


这里我当时是跟transformers逻辑一样，默认值return_dict为True，而paddlenlp基本上所有模型都是False，需要决策一下

改为False吧

JunnYu · 2024-12-31T03:55:17Z

+            if self.gradient_checkpointing and not hidden_states.stop_gradient:
+                layer_outputs = self._gradient_checkpointing_func(


gradient_checkpointing -> recompute，参照paddlenlp的改一下吧

JunnYu · 2024-12-31T03:55:28Z

+        all_self_attentions = () if output_attentions else None
+        all_cross_attentions = () if output_attentions and self.config.add_cross_attention else None
+
+        if self.gradient_checkpointing and self.training:


这里也是

JunnYu · 2024-12-31T03:55:41Z

+        super().__init__()
+        self.config = config
+        self.layer = nn.LayerList([XLMRobertaLayer(config) for _ in range(config.num_hidden_layers)])
+        self.gradient_checkpointing = False


这里也改了吧

改成self.enable_recompute=False

JunnYu · 2024-12-31T03:56:20Z

+    _deprecated_dict = {
+        "key": ".self_attn.q_proj.",
+        "name_mapping": {
+            # common
+            "encoder.layers.": "encoder.layer.",
+            # embeddings
+            "embeddings.layer_norm.": "embeddings.LayerNorm.",
+            # transformer
+            ".self_attn.q_proj.": ".attention.self.query.",
+            ".self_attn.k_proj.": ".attention.self.key.",
+            ".self_attn.v_proj.": ".attention.self.value.",
+            ".self_attn.out_proj.": ".attention.output.dense.",
+            ".norm1.": ".attention.output.LayerNorm.",
+            ".linear1.": ".intermediate.dense.",
+            ".linear2.": ".output.dense.",
+            ".norm2.": ".output.LayerNorm.",
+        },
+    }


这里删了，没有用

JunnYu · 2024-12-31T03:58:42Z

+
+from paddlenlp.transformers.tokenizer_utils import AddedToken
+from paddlenlp.transformers.tokenizer_utils import (
+    PretrainedTokenizer as PPNLPPretrainedTokenizer,


这里不用as直接PretrainedTokenizer

改为相对路径

JunnYu · 2024-12-31T03:58:47Z

+__all__ = ["XLMRobertaTokenizer"]
+
+
+class XLMRobertaTokenizer(PPNLPPretrainedTokenizer):


这里也修改

JunnYu · 2024-12-31T04:00:06Z

auto部分也要加

JunnYu · 2024-12-31T04:01:56Z

+class ModuleUtilsMixin:
+    """
+    A few utilities for `nn.Layer`, to be used as a mixin.
+    """
+
+    # @property
+    # def device(self):
+    #     """
+    #     `paddle.place`: The device on which the module is (assuming that all the module parameters are on the same
+    #     device).
+    #     """
+    #     try:
+    #         return next(self.named_parameters())[1].place
+    #     except StopIteration:
+    #         try:
+    #             return next(self.named_buffers())[1].place
+    #         except StopIteration:
+    #             return paddle.get_device()


这部分的代码加入可能会影响已有的很多模型，得仔细看一下

DrownFish19 · 2024-12-31T04:02:15Z

@@ -0,0 +1,133 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.


这里少一个paddle的copyright

DrownFish19 · 2024-12-31T04:02:52Z

+        classifier_dropout=None,
+        **kwargs,
+    ):
+        kwargs["return_dict"] = kwargs.pop("return_dict", True)


改为False吧

DrownFish19 · 2024-12-31T04:03:12Z

@@ -0,0 +1,1517 @@
+# coding=utf-8


增加paddle的copyright

DrownFish19 · 2024-12-31T04:04:06Z

+from paddle import nn
+from paddle.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
+
+from paddlenlp.transformers.activations import ACT2FN


from paddlenlp 这些都改成相对路径吧

DrownFish19 · 2024-12-31T04:04:49Z

+        super().__init__()
+        self.config = config
+        self.layer = nn.LayerList([XLMRobertaLayer(config) for _ in range(config.num_hidden_layers)])
+        self.gradient_checkpointing = False


改成self.enable_recompute=False

DrownFish19 · 2024-12-31T04:06:10Z

+        Example:
+
+        ```python
+        >>> from ppdiffusers.transformers import AutoTokenizer, XLMRobertaForCausalLM, AutoConfig


同上修改文档

DrownFish19 · 2024-12-31T04:07:34Z

+
+from paddlenlp.transformers.tokenizer_utils import AddedToken
+from paddlenlp.transformers.tokenizer_utils import (
+    PretrainedTokenizer as PPNLPPretrainedTokenizer,


改为相对路径

DrownFish19 · 2024-12-31T04:09:53Z

在PaddleNLP/paddlenlp/transformers/auto文件里增加对应的模型、tokenizer映射

codecov · 2024-12-31T04:16:18Z

Codecov Report

Attention: Patch coverage is 79.34641% with 158 lines in your changes missing coverage. Please review.

Project coverage is 52.39%. Comparing base (dff62a1) to head (473e6e0).
Report is 274 commits behind head on develop.

Files with missing lines	Patch %	Lines
paddlenlp/transformers/xlm_roberta/modeling.py	78.89%	137 Missing ⚠️
paddlenlp/transformers/xlm_roberta/tokenizer.py	75.86%	21 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #9720      +/-   ##
===========================================
- Coverage    53.20%   52.39%   -0.81%     
===========================================
  Files          719      727       +8     
  Lines       115583   115095     -488     
===========================================
- Hits         61493    60304    -1189     
- Misses       54090    54791     +701

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ZHUI · 2025-01-02T09:36:04Z

加两个单测，测试一下，模型初始化，tokenier 加载。

JunnYu · 2025-01-02T10:42:27Z

新增对应的单测脚本

ZHUI · 2025-01-07T03:46:00Z

+    # See all XLM-RoBERTa models at https://huggingface.co/models?filter=xlm-roberta
+]
+
+


缺少 __all__ = [""] 说明一下可以import哪些模型名称

add xlm_roberta in paddlenlp

521a424

JunnYu reviewed Dec 31, 2024

View reviewed changes

DrownFish19 reviewed Dec 31, 2024

View reviewed changes

jie-z-0607 added 3 commits December 31, 2024 16:08

fix1

46ab26e

fix_2

964ac27

fix_configuration

e4c1f12

ZHUI reviewed Jan 7, 2025

View reviewed changes

add_test

473e6e0

jie-z-0607 changed the title ~~add XLM-RoBERTa in paddlenlp~~ Add XLMRoBERTaModel in paddlenlp Jan 8, 2025

ZHUI merged commit 1d74d62 into PaddlePaddle:develop Jan 8, 2025

		if self.gradient_checkpointing and not hidden_states.stop_gradient:
		layer_outputs = self._gradient_checkpointing_func(

		__all__ = ["XLMRobertaTokenizer"]


		class XLMRobertaTokenizer(PPNLPPretrainedTokenizer):

		@@ -0,0 +1,133 @@
		# coding=utf-8
		# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.

		# See all XLM-RoBERTa models at https://huggingface.co/models?filter=xlm-roberta
		]

Conversation

jie-z-0607 commented Dec 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR types

PR changes

Description

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DrownFish19 Dec 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JunnYu commented Dec 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JunnYu Dec 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DrownFish19 Dec 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DrownFish19 commented Dec 31, 2024

Uh oh!

codecov Bot commented Dec 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ZHUI commented Jan 2, 2025

Uh oh!

JunnYu commented Jan 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jie-z-0607 commented Dec 31, 2024 •

edited

Loading

DrownFish19 Dec 31, 2024 •

edited

Loading

JunnYu commented Dec 31, 2024 •

edited

Loading

JunnYu Dec 31, 2024 •

edited

Loading

DrownFish19 Dec 31, 2024 •

edited

Loading

codecov Bot commented Dec 31, 2024 •

edited

Loading