Skip to content

[问题/Issue] 章节3.1:bert位置编码 / Chapter3.1 Encoder-only PLM #166

@zx12908

Description

@zx12908

1. 遇到问题的章节 / Affected Chapter

Chapter3.1

2. 具体问题描述 / Problem Description

bert/gpt/llm模型结构中,针对位置编码其实有很多种版本,感觉需要标明

3. 问题重现材料 / Reproduction Materials

  • rope(llm):QK^T之前
  • bert原版:与transformer中相同的绝对位置编码
  • bert变体:相对位置编码,QK^T之后,softmax之前

确认事项 / Verification

  • 此问题未在过往Issue中被报告过 / This issue hasn't been reported before

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions