[LLM] Add tensor parallel for chatglmv2

[SevenSamon]

PR types

Bug fixes

PR changes

Models

Description

fix chatglmv2 8k

在8k情况下，zero_padding=true, max_sequence_length=8192, src_length=4096

1. 比较 w/o recompute 的显存占用和吞吐情况
   Sharding_parallel_degree=8, sharding=stage3, recompute=True, step=9时, OOM
   Sharding_parallel_degree=8, sharding=stage3, recompute=False, step=1时, OOM
2. 比较 w/o tp、w/o sp 的显存占用和吞吐情况, sharding=4

• Tp=2, sequence parallel = 0, sharding_degree=4,sharding=stage2, recompute=True,
  train_runtime: 293.3676, train_samples_per_second: 0.409, train_steps_per_second: 0.1023, train_loss: 9.233072916666666, progress_or_epoch: 0.0647
  Effective_Tokens_per_second: 3322.318257026338 内存：430G
• TP=2, sequence parallel = 1, sharding_degree=4, sharding=stage2, recompute=True, (padding_length = max_sequence_length)
  train_runtime: 141.7329, train_samples_per_second: 0.8467, train_steps_per_second: 0.2117, train_loss: 9.233072916666666, progress_or_epoch: 0.0647
  Effective_Tokens_per_second: 6876.741628090584 内存：320G

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

Attention: Patch coverage is 43.38843% with 137 lines in your changes missing coverage. Please review.

Project coverage is 53.79%. Comparing base (48820cb) to head (42bf444).
Report is 261 commits behind head on develop.

Files with missing lines	Patch %	Lines
paddlenlp/transformers/chatglm_v2/modeling.py	43.56%	136 Missing ⚠️
...p/experimental/transformers/chatglm_v2/modeling.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #9014      +/-   ##
===========================================
+ Coverage    53.34%   53.79%   +0.45%     
===========================================
  Files          650      652       +2     
  Lines       105414   104710     -704     
===========================================
+ Hits         56230    56329      +99     
+ Misses       49184    48381     -803     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[DrownFish19]

LGTM

[ZHUI]

这个问题出现的多吗？