[Bug] Recurring matrix dimensions mismatch issue during GRPO training on 2 Nvidia A100s through GCP.

[prakritishetty]

**```
torch._dynamo.exc.TorchRuntimeError: Dynamo failed to run FX node with fake tensors: call_function <built-in method matmul of type object at 0x77cd34ddba20>(*(GradTrackingTensor(lvl=1, value=
FakeTensor(..., device='cuda:0', size=(1, s17, s6), dtype=torch.bfloat16,
requires_grad=True)
), GradTrackingTensor(lvl=1, value=
FakeTensor(..., device='cuda:0', size=(2880, 201088), dtype=torch.bfloat16)
)), **{}): got RuntimeError('a and b must have same reduction dim, but got [s17, s6] X [2880, 201088].')

Enviroment: 2 Nvidia 80G A100s on a single GCP VM - ssh through vscode. 

[danielhanchen]

@prakritishetty Please rotate your HF token (you missed one) :) I editted your issue

Hmm could you try at the very top before any Unsloth import:

import os
os.environ["UNSLOTH_ENABLE_LOGGING"] = "1"

maybe you might get better logs.

Then if you want, try:

import os
os.environ["UNSLOTH_ENABLE_LOGGING"] = "1"
os.environ["UNSLOTH_COMPILE_DISABLE"] = "1"

And if that fails, try:

import os
os.environ["UNSLOTH_ENABLE_LOGGING"] = "1"
os.environ["UNSLOTH_COMPILE_DISABLE"] = "1"
os.environ["UNSLOTH_RETURN_LOGITS"] = "1"

[Datta0]

Hey @prakritishetty did you by any chance set any environment variables like os.environ["UNSLOTH_COMPILE_DISABLE"] = "1" as this might be a potential cause?

Also did you by any chance modify the tokenizer aka add new tokens?