Slow training

[chan-yuu]

I am training the model with transfuser, but it trains very slowly:

(navsim) user@user-Super-Server:~/Documents/cyun/navsim/DiffusionDriveV2$ python navsim/planning/script/run_training.py \
>     agent=transfuser_agent \
>     experiment_name=transfuser_training \
>     train_test_split=navtrain \
>     use_cache_without_dataset=true \
>     force_cache_computation=false

Is there any parameter that can improve the efficiency of training? I am using the A800, memory and memory are very sufficient.

Epoch 0:   0%|                                                                              | 0/1330 [00:00<?, ?it/s][W reducer.cpp:1300] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
Epoch 0:   1%|▍                                  | 16/1330 [01:04<1:27:55,  0.25it/s, v_num=0, train/loss_step=77.00]