Skip to content

Conversation

@0x45f
Copy link
Contributor

@0x45f 0x45f commented May 21, 2024

PR Category

Others

PR Types

Bug fixes

Description

Pcard-67164
修复bert在PIR AMP下暴露框架问题:

  • fused_attention_grad和fused_feedforward_grad两个反向算子没有配置forward字段,导致在append_backward时会跳过这两个反向算子,从而导致反向program中没有这两个op
  • op_gen.py中有的地方漏掉了对于sp算子的处理,导致代码生成出错
  • fused_attention_grad在op代码生成时不符合之前原有设想,需要在validate_backward_outputs函数中跳过对于fused_attention_grad的检查,因为对fused_attention_grad来说len(backward_outputs) <= len(forward_inputs)
  • fused_attention_grad比较特殊,对fused_attention_vjp进行了手写处理
  • fused_feedforward_grad输出参数顺序有问题进行了调整

@paddle-bot
Copy link

paddle-bot bot commented May 21, 2024

你的PR提交成功,感谢你对开源项目的贡献!
请关注后续CI自动化测试结果,详情请参考Paddle-CI手册
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@0x45f 0x45f changed the title Fix some error for bert amp [PIR AMP]Fix some error for bert amp May 23, 2024
return vjp_res;
}

std::vector<std::vector<paddle::Tensor>> fused_attention_vjp(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个看起来好像可以codegen?

@0x45f 0x45f merged commit c942ecb into PaddlePaddle:develop May 23, 2024
@0x45f 0x45f deleted the fix-bert-amp branch May 23, 2024 06:57
co63oc pushed a commit to co63oc/Paddle that referenced this pull request May 23, 2024
chen2016013 pushed a commit to chen2016013/Paddle that referenced this pull request May 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants