dune build -w restricting job number for no reason

[Alizter]

Problem description

There seems to be an issue with dune build -w. It can be observably slow due to a restriction of jobs. This can be seen as slow, since stopping watch mode and doing a fresh dune build will be much faster. The issue seems to be during the rule finding phase.

Here is a demonstration GIF of this behaviour. Details of what is happening are below:

1. dune build -w is run, and it works as expected. Most of the time is spent rule finding, but due to cache and everything already being built, it finishes quickly.
2. Next whilst watch mode is still running, an .ml file early in the dependency tree is changed, triggering a rebuild. Immediately you can observe the number of jobs drop to a low count. What is dune doing here?
3. To demonstrate that this is an issue with watch mode, a fresh dune build is triggered, and as expected everything builds normally and quickly.

There are some key details with this setup. This is the main Coq repo being changed, and importantly there is ml code being edited which are later being depended on by dune Coq stanzas. Retrying the same test, but this time with the @check target (building only ml code) results in no observable slow down, leading me to believe that this is an issue with the interaction of rule finding and the Coq rules.

Reproduction

1. Clone the https://github.com/coq/coq repo.
2. Do dune build -w. This will take a while but once it starts building .v files (see from --display=short or --verbose) you can do the next step.
3. Edit an ml file, for example kernel/declareops.ml, you can even drop let _hello = "world" inside.
4. Observe that the rebuild will now restrict itself to a low job count.
5. Stopping watch mode and rebuilding will speed up the build again.

Specifications

• Version of dune (output of dune --version): 3.0.3
• Version of ocaml (output of ocamlc --version) 4.12.1
• Operating system (distribution and version): Ubuntu

cc @rgrinberg

[ejgallego]

@Alizter if you do dune build -w theories/Reals/Real.vo for example, that is to say, with a concrete target, do you see the same behavior?

[Alizter]

@ejgallego Yes.

[rgrinberg]

From the meeting: there must be some mutable state remaining in dune that is causing this. We need to scan the code base once more for mutable state and convert it to Memo

[jchavarri]

I am seeing the same behavior described in the issue for a mid-large size Melange build. After watch mode restarts when the first change happens, the jobs count drops to 1, with very short peaks of 2-3 jobs. At Ahrefs, we use cpus with large parallelism, and most part of the build can be done using 255 parallel jobs, so the performance impact is severe. We would like to move away from external watch tools like watchexec, as they interfere with the recently added lock to prevent multiple build commands (#6360), but the jobs restricting issue would prevent the migration to Dune watch.

We need to scan the code base once more for mutable state and convert it to Memo

Are there any previous examples in PRs to show this type of conversion and how they look like? Also, is there some way to debug the dep graph information? I understand that for a set of rules that can be executed in parallel to become basically a sequence, each rule must at some point start depending on the previous one somehow, starting from the second and subsequent builds?

[Alizter]

@jchavarri Does this only happen with Melange builds? Have you observed it with OCaml? If not, then that might be an indicator to what is going wrong since Melange and Coq do some similar things.

[jchavarri]

It also happens with OCaml.

Some more observations:

• During the time where it gets stucked with 1 job, the left number shows a very low number as well, and the % remains always at 99%, e.g.: Done: 99% (908/913, 5 left) (jobs: 1). In the gif from the issue description, it shows 1 left. It looks like the queue only contains the most immediate rules, and it keeps adding new ones constantly after executing the current one?
• I can reproduce the issue when updating both ml files, but also other kinds, like atd files
• However, the issue does not happen when modifying any ml file. Some ml files trigger a parallelized build, while others do not. I have not yet figured out any patterns, but the behavior is consistent: if some file change breaks parallelization, it will always do so.

[rgrinberg]

However, the issue does not happen when modifying any ml file. Some ml files trigger a parallelized build, while others do not. I have not yet figured out any patterns, but the behavior is consistent: if some file change breaks parallelization, it will always do so.

Thanks for the info. Could you find out the following:

1. The .ml flies that do not cause this problem have mlis?
2. How many other modules depend on the problematic .ml files?

[jchavarri]

The .ml flies that do not cause this problem have mlis?

I did not find any relation between ml files having mli or not.

How many other modules depend on the problematic .ml files?

There are hundreds of modules that depend on the .ml files. I noticed that rather than specific ml files being problematic, it seems to be an issue across the library boundary (maybe).

I was able to find a simplified repro using synthetic code, that I put up in https://github.com/jchavarri/dune_repro_issue_5549. Instructions from the readme copied below:

Dune repro issue 5549

1. Install switch: make create-switch

2. Run Dune in watch mode: dune build -w @all, notice how the build leverages all cores in cpu.

3. Modify chunk2/dir_2_1/m2_1_1_1.ml and mli, notice how the build finishes very fast, even if a few modules depend on it.

4. Modify chunk1/dir_1_1/m1_1_1_1.ml and mli, notice how the build only executes 1 job at a time.

[jchavarri]

Some added notes:

• I can reproduce on both macOS and linux
• the number of jobs is mostly 1, but can go to 2, 3 or larger numbers in short times
• I looked for refs through the codebase and there are a lot. Many of them in stdune (not sure if they could be affecting the build indirectly). But one that caught my attention is this:

dune/src/dune_engine/build_system.ml

Line 221 in 8d88ee8

let pending_file_targets = ref Path.Build.Set.empty

Could it be related somehow? Is there a way to debug this to find the problem?

[jchavarri]

Another note: the bug is reproducible on 3.0.2 but not on 2.9.3 (both installed from opam). Trying to figure out now the exact commit where it regressed.

[jchavarri]

I can pin the regression down to this commit:

fb3c71d.

[rgrinberg]

Do you mind trying the following:

1. Checkout the commit immediately before fb3c71d

2. Set DUNE_WATCHING_MODE_INCREMENTAL=true

3. Try to reproduce the bug.

If so, then I would suggest to bisect further but with this environment variable switched on.