Migrate benchmark/MLOps GPU pipeline to Terraform/OpenTofu + cloud provider study (Alibaba vs DigitalOcean)

[miguelgfierro]

Summary

Move the GPU compute used by the benchmark/MLOps pipeline (currently run on ad-hoc hardware) to Infrastructure-as-Code using Terraform / OpenTofu, so benchmark environments are reproducible, disposable, and cost-controlled. As part of provider selection, this issue records a study comparing Alibaba Cloud vs DigitalOcean GPU offerings.

Motivation

• The MovieLens benchmark (examples/06_benchmarks/movielens.ipynb) currently depends on whatever machine happens to be available (last run: RTX 5090 Laptop, 24 vCPU, 30 GB RAM). Not reproducible.
• Manual GPU boxes are easy to leave running and bleed cost (both providers keep billing while powered off — you must destroy the instance to stop the meter).
• IaC makes each run spin up → run → destroy, fully declarative and repeatable.

Proposed change

• Add a terraform/ (or infra/) module that provisions a GPU instance and tears it down, using OpenTofu-compatible providers.
• Both target providers have first-class OpenTofu support via official providers: aliyun/alicloud and digitalocean/digitalocean (both in the OpenTofu registry).
• Wire the benchmark run into the provisioned box: setup env → execute notebook → collect metrics → destroy.

Cloud provider study: Alibaba Cloud vs DigitalOcean

IaC support

Both support Terraform and OpenTofu via official providers. Caveat: DigitalOcean's A100 lives on the Paperspace side, which is not covered by the official digitalocean provider (weak IaC story — only a separate early-stage Paperspace/paperspace provider, not in the OpenTofu registry). Core GPU Droplets (H100/H200/L40S/RTX Ada) are fully covered.

Billing

Both bill per-second (DigitalOcean: 60-second minimum), no hour rounding. Both keep charging while powered off → must destroy to stop the meter.

GPU comparison (inference / light-training class)

	Alibaba T4 (gn6i)	DigitalOcean RTX 4000 Ada
Architecture / memory	Turing 2018 / 16 GB	Ada 2023 / 20 GB
On-demand price	~$1.20/hr	$0.76/hr

• No true "same GPU" for training: the T4 exists only on Alibaba; DigitalOcean has no T4. For training, the only NVIDIA overlap is the A100 (Alibaba ~$2.40/hr vs DigitalOcean/Paperspace ~$3.18/hr); H100/H200 are DigitalOcean-only.
• Estimated cost of one full benchmark run (10–15 min wall on ML-100k): **$0.15–0.30** — negligible either way; RTX 4000 Ada is ~35–40% cheaper and faster than the T4.
• The real cost driver is idle time, not run time. Note ~1/3 of the pipeline is CPU/Spark-bound (ALS, SAR, BPR), so vCPU count matters as much as the GPU for ML-100k.

Tasks

• Add an OpenTofu module provisioning a GPU instance (parametrized by provider).
• Add teardown/destroy step + guard against orphaned instances.
• Script: setup env → run benchmark notebook → export metrics → destroy.
• Decide provider/GPU (recommend DigitalOcean RTX 4000 Ada for cost/reproducibility; A100 if larger training is needed).
• Document usage in examples/06_benchmarks/README.md.

References

• Alibaba GPU/T4 pricing
• DigitalOcean GPU Droplet pricing
• OpenTofu registry — alicloud · digitalocean
• Alibaba per-second billing · DigitalOcean billing

[miguelgfierro]

FYI @SimonYansenZhao

[miguelgfierro]

@SimonYansenZhao I think something important is that we should be able to cut and destroy a MLOps process and kill the VM if there is a new commit. Otherwise, we will get charged by all the runs when we only care about the latest one.