Vast.ai pricing

[gelim]

Hey, thanks @karpathy for this marvel.
When seeing that you use Lambda as the cloud GPU provider, there is an alternative vast.ai that could help lower down a bit the total price with following rates:

• H100 ~17$/hour
• H200 ~22$/hour
• B200 ~24$/hour: need to assess speed gain

[gelim]

spinning on 8xH200 gives that kind of perf during base_train:

step 00241/21400 (1.13%) | loss: 4.132522 | lrm: 1.00 | dt: 494.57ms | tok/sec: 530,039 | mfu: 46.78 | total time: 1.92m
step 00242/21400 (1.13%) | loss: 4.124904 | lrm: 1.00 | dt: 499.75ms | tok/sec: 524,550 | mfu: 46.30 | total time: 1.93m

[rfernand2]

Thanks - great to have a reference for 8xH200. But I am seeing tok/sec of 1,000,000 on 8xH100 on LambdaLabs Cloud. Thoughts?

[dcs-scd]

what batch size did you use?

[rfernand2]

The default, device_batch_size=32