[OPT] Run test in lower precision on GPU #17353

patrickvonplaten · 2022-05-19T14:23:27Z

What does this PR do?

Let's run the test only in half precision. There might have been a bug because fp16 weights are loaded into fp32. Let's make sure everything stays in fp16.

Also if it fails again, we'll see a better error message this time.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

HuggingFaceDocBuilderDev · 2022-05-19T14:55:26Z

The documentation is not available anymore as the PR was closed or merged.

patrickvonplaten · 2022-05-19T15:51:42Z

src/transformers/models/opt/modeling_opt.py

-            self.scale_grad_by_freq,
-            self.sparse,
-        )
+    def __init__(self, num_embeddings: int, embedding_dim: int):


Cleaned this up a bit. Think we should try to align it as much as possible to Bart here

Thanks :) , I have to update the tf code based on that I think

patrickvonplaten · 2022-05-19T15:58:32Z

tests/models/opt/test_modeling_opt.py

 @require_torch
 class OPTModelIntegrationTests(unittest.TestCase):
    @slow
+    @unittest.skipIf(torch_device != "cpu", "Cannot make deterministic on GPU")


Getting different logits results on GPU depending on PyTorch version (1.10+cu11.0 vs. 1.11+cu11.4) and results also differ between CPU and GPU. Only on CPU it seems to be deterministic.

It's not because the weights are saved & loaded in FP16 - checked that the same happens when weights are stored in fp32 and loaded in fp32.
It's across multiple checkpoints (not just 350m, but also 125m).
The differences start to creep in in the first linear projection matrix project_in_dim for the 350m model and the first attention layer for 125m.

Upgrading precision to 1e-2 so that tests pass on GPU - think that's the best we can do

Interesting observation:

BART has the same problem, but the differences are not as flagrant in the end

BERT does not have this problem at all. Results are exactly the same.

cc @stephenroller @stas00 @sgugger @LysandreJik

getting logits to exact match is very difficult at times. Perhaps using a much longer input and checking that softmax output matches - after all logits fluctuations don't matter at the end but 100% softmax match does.

if you use a very high tolerance then it becomes questionable whether the test is doing anything.

ydshieh

I will allow 1e-2 for now 😄

ydshieh · 2022-05-19T16:21:50Z

tests/models/opt/test_modeling_opt.py

 class OPTModelIntegrationTests(unittest.TestCase):
    @slow
    def test_inference_no_head(self):
+        # model is not deterministic on GPU, not sure why


(some) non-deterministic is mentioned
https://pytorch.org/docs/stable/notes/randomness.html

The diff 1e-2 is still unpleasantly large though.

Actually my comment is not great - the model is deterministic on GPU, but it's just that results differ depending on the CUDA vesion and CUDA != CPU

tests/models/opt/test_modeling_opt.py

sgugger

Thanks for fixing!

sgugger · 2022-05-19T17:42:29Z

tests/models/opt/test_modeling_opt.py

+        atol = 1e-2 if torch_device != "cpu" else 1e-3
+        assert_tensors_close(output[0, :3, :3], expected_slice, atol=atol)


Ouch, that's some difference.

commit 5419205 Author: Patrick von Platen <[email protected]> Date: Thu May 19 23:46:26 2022 +0200 [Test OPT] Add batch generation test opt (huggingface#17359) * up * up commit 48c2269 Author: ddobokki <[email protected]> Date: Fri May 20 05:42:44 2022 +0900 Fix bug in Wav2Vec2 pretrain example (huggingface#17326) commit 5d6feec Author: Nathan Dahlberg <[email protected]> Date: Thu May 19 16:21:19 2022 -0400 fix for 17292 (huggingface#17293) commit 518bd02 Author: Patrick von Platen <[email protected]> Date: Thu May 19 22:17:02 2022 +0200 [Generation] Fix Transition probs (huggingface#17311) * [Draft] fix transition probs * up * up * up * make it work * fix * finish * update commit e8714c0 Author: Patrick von Platen <[email protected]> Date: Thu May 19 22:15:36 2022 +0200 [OPT] Run test in lower precision on GPU (huggingface#17353) * [OPT] Run test only in half precision * up * up * up * up * finish * fix on GPU * Update tests/models/opt/test_modeling_opt.py commit 2b28229 Author: Nicolas Patry <[email protected]> Date: Thu May 19 20:28:12 2022 +0200 Adding `batch_size` test to QA pipeline. (huggingface#17330) commit a4386d7 Author: Nicolas Patry <[email protected]> Date: Thu May 19 10:29:16 2022 +0200 [BC] Fixing usage of text pairs (huggingface#17324) * [BC] Fixing usage of text pairs The BC is actually preventing users from misusing the pipeline since users could have been willing to send text pairs and the pipeline would instead understand the thing as a batch returning bogus results. The correct usage of text pairs is preserved in this PR even when that makes the code clunky. Adds support for {"text":..,, "text_pair": ...} inputs for both dataset iteration and more explicit usage to pairs. * Updating the doc. * Update src/transformers/pipelines/text_classification.py Co-authored-by: Sylvain Gugger <[email protected]> * Update src/transformers/pipelines/text_classification.py Co-authored-by: Sylvain Gugger <[email protected]> * Update tests/pipelines/test_pipelines_text_classification.py Co-authored-by: Lysandre Debut <[email protected]> * quality. Co-authored-by: Sylvain Gugger <[email protected]> Co-authored-by: Lysandre Debut <[email protected]> commit 3601aa8 Author: Stas Bekman <[email protected]> Date: Wed May 18 16:00:47 2022 -0700 [tests] fix copy-n-paste error (huggingface#17312) * [tests] fix copy-n-paste error * fix commit 1b20c97 Author: Yih-Dar <[email protected]> Date: Wed May 18 21:49:08 2022 +0200 Fix ci_url might be None (huggingface#17332) * fix * Update utils/notification_service.py Co-authored-by: Lysandre Debut <[email protected]> Co-authored-by: ydshieh <[email protected]> Co-authored-by: Lysandre Debut <[email protected]> commit 6aad387 Author: Yih-Dar <[email protected]> Date: Wed May 18 21:26:44 2022 +0200 fix (huggingface#17337) Co-authored-by: ydshieh <[email protected]> commit 1762ded Author: Zachary Mueller <[email protected]> Date: Wed May 18 14:17:40 2022 -0400 Fix metric calculation in examples and setup tests to run on multi-gpu for no_trainer scripts (huggingface#17331) * Fix length in no_trainer examples * Add setup and teardown * Use new accelerator config generator to automatically make tests able to run based on environment commit 6e195eb Author: Jader Martins <[email protected]> Date: Wed May 18 14:18:43 2022 -0300 docs for typical decoding (huggingface#17186) Co-authored-by: Jader Martins <[email protected]> commit 060fe61 Author: Yih-Dar <[email protected]> Date: Wed May 18 19:07:48 2022 +0200 Not send successful report (huggingface#17329) * send report only if there is any failure Co-authored-by: ydshieh <[email protected]> commit b3b9f99 Author: Yih-Dar <[email protected]> Date: Wed May 18 17:57:23 2022 +0200 Fix test_t5_decoder_model_past_large_inputs (huggingface#17320) Co-authored-by: ydshieh <[email protected]> commit 6da76b9 Author: Jingya HUANG <[email protected]> Date: Wed May 18 17:52:13 2022 +0200 Add onnx export cuda support (huggingface#17183) Co-authored-by: Lysandre Debut <[email protected]> Co-authored-by: lewtun <[email protected]> commit adc0ff2 Author: NielsRogge <[email protected]> Date: Wed May 18 17:47:18 2022 +0200 Add CvT (huggingface#17299) * Adding cvt files * Adding cvt files * changes in init file * Adding cvt files * changes in init file * Style fixes * Address comments from code review * Apply suggestions from code review Co-authored-by: Sylvain Gugger <[email protected]> * Format lists in docstring * Fix copies * Apply suggestion from code review Co-authored-by: AnugunjNaman <[email protected]> Co-authored-by: Ayushman Singh <[email protected]> Co-authored-by: Niels Rogge <[email protected]> Co-authored-by: Sylvain Gugger <[email protected]> commit 4710702 Author: Sylvain Gugger <[email protected]> Date: Wed May 18 10:46:40 2022 -0400 Fix style commit 5fdb54e Author: mraunak <[email protected]> Date: Wed May 18 10:39:02 2022 -0400 Add Information Gain Filtration algorithm (huggingface#16953) * Add information gain filtration algorithm * Complying with black requirements * Added author * Fixed import order * flake8 corrections Co-authored-by: Javier Turek <[email protected]> commit 91ede48 Author: Kamal Raj <[email protected]> Date: Wed May 18 19:59:53 2022 +0530 Fix typo (huggingface#17328) commit fe28eb9 Author: Yih-Dar <[email protected]> Date: Wed May 18 16:06:41 2022 +0200 remove (huggingface#17325) Co-authored-by: ydshieh <[email protected]> commit 2cb2ea3 Author: Nicolas Patry <[email protected]> Date: Wed May 18 16:06:24 2022 +0200 Accepting real pytorch device as arguments. (huggingface#17318) * Accepting real pytorch device as arguments. * is_torch_available. commit 1c9d1f4 Author: Nicolas Patry <[email protected]> Date: Wed May 18 15:46:12 2022 +0200 Updating the docs for `max_seq_len` in QA pipeline (huggingface#17316) commit 60ad734 Author: Patrick von Platen <[email protected]> Date: Wed May 18 15:08:56 2022 +0200 [T5] Fix init in TF and Flax for pretraining (huggingface#17294) * fix init * Apply suggestions from code review * fix * finish * Update src/transformers/modeling_tf_utils.py Co-authored-by: Sylvain Gugger <[email protected]> Co-authored-by: Sylvain Gugger <[email protected]> commit 7ba1d4e Author: Joaq <[email protected]> Date: Wed May 18 09:23:47 2022 -0300 Add type hints for ProphetNet (Pytorch) (huggingface#17223) * added type hints to prophetnet * reformatted with black * fix bc black misformatted some parts * fix imports * fix imports * Update src/transformers/models/prophetnet/configuration_prophetnet.py Co-authored-by: Matt <[email protected]> * update OPTIONAL type hint and docstring Co-authored-by: Matt <[email protected]> commit d6b8e9c Author: Carl <[email protected]> Date: Wed May 18 01:07:43 2022 +0200 Add trajectory transformer (huggingface#17141) * Add trajectory transformer Fix model init Fix end of lines for .mdx files Add trajectory transformer model to toctree Add forward input docs Fix docs, remove prints, simplify prediction test Apply suggestions from code review Co-authored-by: Sylvain Gugger <[email protected]> Apply suggestions from code review Co-authored-by: Lysandre Debut <[email protected]> Co-authored-by: Sylvain Gugger <[email protected]> Update docs, more descriptive comments Apply suggestions from code review Co-authored-by: Sylvain Gugger <[email protected]> Update readme Small comment update and add conversion script Rebase and reformat Fix copies Fix rebase, remove duplicates Fix rebase, remove duplicates * Remove tapex * Remove tapex * Remove tapex commit c352640 Author: Patrick von Platen <[email protected]> Date: Wed May 18 00:34:31 2022 +0200 fix (huggingface#17310) commit d9050dc Author: Cesare Campagnano <[email protected]> Date: Tue May 17 23:44:37 2022 +0200 [LED] fix global_attention_mask not being passed for generation and docs clarification about grad checkpointing (huggingface#17112) * [LED] fixed global_attention_mask not passed for generation + docs clarification for gradient checkpointing * LED docs clarification Co-authored-by: Patrick von Platen <[email protected]> * [LED] gradient_checkpointing=True should be passed to TrainingArguments Co-authored-by: Patrick von Platen <[email protected]> * [LED] docs: remove wrong word Co-authored-by: Patrick von Platen <[email protected]> * [LED] docs fix typo Co-authored-by: Patrick von Platen <[email protected]> Co-authored-by: Patrick von Platen <[email protected]> commit bad3583 Author: Jean Vancoppenolle <[email protected]> Date: Tue May 17 23:42:14 2022 +0200 Add support for pretraining recurring span selection to Splinter (huggingface#17247) * Add SplinterForSpanSelection for pre-training recurring span selection. * Formatting. * Rename SplinterForSpanSelection to SplinterForPreTraining. * Ensure repo consistency * Fixup changes * Address SplinterForPreTraining PR comments * Incorporate feedback and derive multiple question tokens per example. * Update src/transformers/models/splinter/modeling_splinter.py Co-authored-by: Patrick von Platen <[email protected]> * Update src/transformers/models/splinter/modeling_splinter.py Co-authored-by: Patrick von Platen <[email protected]> Co-authored-by: Jean Vancoppenole <[email protected]> Co-authored-by: Tobias Günther <[email protected]> Co-authored-by: Tobias Günther <[email protected]> Co-authored-by: Patrick von Platen <[email protected]> commit 0511305 Author: Yih-Dar <[email protected]> Date: Tue May 17 18:56:58 2022 +0200 Add PR author in CI report + merged by info (huggingface#17298) * Add author info to CI report * Add merged by info * update Co-authored-by: ydshieh <[email protected]> commit 032d63b Author: Sylvain Gugger <[email protected]> Date: Tue May 17 12:56:24 2022 -0400 Fix dummy creation script (huggingface#17304) commit 986dd5c Author: Sylvain Gugger <[email protected]> Date: Tue May 17 12:50:14 2022 -0400 Fix style commit 38ddab1 Author: Karim Foda <[email protected]> Date: Tue May 17 09:32:12 2022 -0700 Doctest longformer (huggingface#16441) * Add initial doctring changes * make fixup * Add TF doc changes * fix seq classifier output * fix quality errors * t * swithc head to random init * Fix expected outputs * Update src/transformers/models/longformer/modeling_longformer.py Co-authored-by: Yih-Dar <[email protected]> Co-authored-by: Yih-Dar <[email protected]> commit 10704e1 Author: Patrick von Platen <[email protected]> Date: Tue May 17 18:20:36 2022 +0200 [Test] Fix W2V-Conformer integration test (huggingface#17303) * [Test] Fix W2V-Conformer integration test * correct w2v2 * up commit 28a0811 Author: regisss <[email protected]> Date: Tue May 17 17:58:14 2022 +0200 Improve mismatched sizes management when loading a pretrained model (huggingface#17257) - Add --ignore_mismatched_sizes argument to classification examples - Expand the error message when loading a model whose head dimensions are different from expected dimensions commit 1f13ba8 Author: Patrick von Platen <[email protected]> Date: Tue May 17 15:48:23 2022 +0200 correct opt (huggingface#17301) commit 349f1c8 Author: Matt <[email protected]> Date: Tue May 17 14:36:23 2022 +0100 Rewrite TensorFlow train_step and test_step (huggingface#17057) * Initial commit * Better label renaming * Remove breakpoint before pushing (this is your job) * Test a lot more in the Keras fit() test * make fixup * Clarify the case where we flatten y dicts into tensors * Clarify the case where we flatten y dicts into tensors * Extract label name remapping to a method commit 651e48e Author: Matt <[email protected]> Date: Tue May 17 14:14:17 2022 +0100 Fix tests of mixed precision now that experimental is deprecated (huggingface#17300) * Fix tests of mixed precision now that experimental is deprecated * Fix mixed precision in training_args_tf.py too commit 6d21142 Author: SaulLu <[email protected]> Date: Tue May 17 14:33:13 2022 +0200 fix retribert's `test_torch_encode_plus_sent_to_model` (huggingface#17231)

* [OPT] Run test only in half precision * up * up * up * up * finish * fix on GPU * Update tests/models/opt/test_modeling_opt.py

[OPT] Run test only in half precision

db27c99

patrickvonplaten requested review from ArthurZucker and ydshieh May 19, 2022 14:24

up

77b2f6e

up

a533e06

ArthurZucker approved these changes May 19, 2022

View reviewed changes

patrickvonplaten added 3 commits May 19, 2022 11:32

up

cccd907

up

6002a65

finish

f72ae0d

patrickvonplaten commented May 19, 2022

View reviewed changes

fix on GPU

3db3ac2

patrickvonplaten changed the title ~~[OPT] Run test only in half precision~~ [OPT] Run test in lower precision on GPU May 19, 2022

ydshieh approved these changes May 19, 2022

View reviewed changes

patrickvonplaten commented May 19, 2022

View reviewed changes

tests/models/opt/test_modeling_opt.py Outdated Show resolved Hide resolved

Update tests/models/opt/test_modeling_opt.py

8f1cc48

sgugger approved these changes May 19, 2022

View reviewed changes

patrickvonplaten merged commit e8714c0 into main May 19, 2022

patrickvonplaten deleted the fix_opt_only_half branch May 19, 2022 20:15

		atol = 1e-2 if torch_device != "cpu" else 1e-3
		assert_tensors_close(output[0, :3, :3], expected_slice, atol=atol)

[OPT] Run test in lower precision on GPU #17353

[OPT] Run test in lower precision on GPU #17353

Conversation

patrickvonplaten commented May 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented May 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stas00 May 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ydshieh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sgugger left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

patrickvonplaten commented May 19, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented May 19, 2022 •

edited

Loading

stas00 May 19, 2022 •

edited

Loading