cmd/go: critical path scheduling

[dvyukov]

Currently go tool schedules work in topological order, this leads to suboptimal
parallelization. It's possible that towards end, very few work items are available
leading to idle processors.

Critical path scheduling gives much better results in this respect:
http://en.wikipedia.org/wiki/Critical_path_method

Here is a prototype CL:
https://golang.org/cl/13235046/diff/3001/src/cmd/go/build.go
I've observed up to 2x 'go test -a std' speedup on a 32-way machine.

We can also assign meaningful weights to work nodes: e.g. test > link > compile.
It would also be super useful to somehow assign weights to test nodes proportional to
test run time (e.g. runtime tests take significantly more than than most packages). This
will have significant effect even on low-core machines (as runtime tests will start
first).

[rsc]

Comment 1:

Please send CL after Go 1.4 is out.

Labels changed: added release-go1.5, removed release-none.

Status changed to Accepted.

[josharian]

@dvyukov are you planning to migrate that CL to gerrit and mail? Does it speed up make.bash as well?

[dvyukov]

Is anybody planning to review it to +2 state?

[josharian]

I promise to review it thoroughly (although I'll be in and out until late March). And rsc asked for it to be mailed. If you don't want to bother, do you mind if I take it over?

The existing CL looks possibly incomplete. assignPriorities contains (lines 648-649):

if a.priority < a1.priority+1 {
}

[dvyukov]

Take over it.

That if should be:

if a.priority < a1.priority+a.weight {
a.priority = a1.priority+a.weight
}

Also weight must be more sensible, that is, represent real cost of action. Something along the lines of:
link > compile
run test > link
Fake instant actions should have weight << everything else.
The goal here is to avoid serial tail of actions that run one after another delaying whole thing.

Ideally 'run test' weights represent actual execution time, so that runtime test starts first rather then starts late and delays all.bash. This may be possible in 'go test' continuos mode that monitors filesystem changes. But that's of course not a part of this change.

[minux]

And compiling anything involving cgo should be even heavier weight than compiling any other packages. If we reorder tests, do we need to preserve the order of showing the results? I think we tried hard to show a sorted result for "go test -short std".

[dvyukov]

And compiling anything involving cgo should be even heavier weight than
compiling any other packages.

good idea

If we reorder tests, do we need to preserve the order of showing the
results?

absolutely
but that is already ensured by chaining printing actions with dependencies, otherwise they would be printed out of order even today with p>1
so you don't need to do anything special here