Allow fractional jobs to automatically scale by number of CPUs (200USD Bounty)

[lihaoyi]

---

From the maintainer Li Haoyi: I'm putting a 200USD bounty on this issue, payable by bank transfer on a merged PR implementing this.

---

so -j 0.5 on a 10 CPU machine would be the same as -j 5, on a 4 CPU machine the same as -j 2, etc. -j 1.0 would be the current default, and -j 2.0 would mean twice as many jobs as CPUs (e.g. if they're mostly IO bound).

I've wanted -j 0.5 myself to use in Mill's own CI, because Mill tests are very heavy and use more than one thread per task, and it would be nice if the same -j config would just do the right thing on different machines with different amounts of available compute

[lefou]

Nice idea.

[wb14123]

Shall we use another flag for this? It's confusing that -j 1.0 is not equal to -j 1.

[lihaoyi]

how about -j 0.5n or -j 0.5*HOST_CPUS (what baze uses)?

[lihaoyi]

Bazel supports basic arithmetic and allows things like HOST_CPUS-1 as well

[wb14123]

Yeah I think we can take what Brazel uses, specifically in this doc:

--local_{ram,cpu}_resources resources or resource expression
These options specify the amount of local resources (RAM in MB and number of CPU logical cores) that Bazel can take into consideration when scheduling build and test activities to run locally. They take an integer, or a keyword (HOST_RAM or HOST_CPUS) optionally followed by [-|float] (for example, --local_cpu_resources=2, --local_ram_resources=HOST_RAM.5, --local_cpu_resources=HOST_CPUS-1). The flags are independent; one or both may be set. By default, Bazel estimates the amount of RAM and number of CPU cores directly from the local system's configuration.

It only allows keyword at the beginning and only supports * and -, which makes parsing not too hard.

[lihaoyi]

Yeah don't even need a real parser, pattern matching on case s"HOST_CPUS*$n" => would be enough

[wb14123]

Oh that's cool! I didn't know you can do that in Scala. I can try to implement this feature.

[lefou]

In Maven it is -T n for using n threads and -T nC for using n threads per CPU core, which I find nice.

On a 16 core cpu, these three command would all mean the same:

mill -j 0
mill -j 16
mill -j 1C

It's basically what @lihaoyi suggested in the first comment, but with a C suffix for "per core".

[lihaoyi]

@wb14123 go for it. I don't mind either the maven way or the bazel way, either works as long as we document it

[wb14123]

I'm wondering if HOST_CPU-1 can be useful for some cases? It's basically using most of the cores but leave 1 core for some other tasks in the OS. If we want to make it shorter we can support things like 0.5C and C-1.

[lefou]

I'm not used to bazel, but I find the term HOST_CPU misleading, since my machine has only one cpu but 16 cores.

[wb14123]

Hmm, then I think we can support the format like maven 0.5C, but also add support for things like C-1.