Distributed test suite: add an additional use of the poll_while function

[DilumAluthge]

Follow-up to #42499

---

This pull request is intended to fix the following test failure, which I am seeing very rarely in the tester_linux64_mt Buildkite job:

Distributed                         (1) |        started at 2021-10-22T10:10:08.433
Test Failed at /cache/build/amdci7-2/julialang/julia-master/julia-609a4a0a1e/share/julia/stdlib/v1.8/Distributed/test/distributed_exec.jl:270
  Expression: remotecall_fetch((k->begin
                haskey(Distributed.PGRP.refs, k)
            end), wid1, rrid) == true
   Evaluated: false == true
ERROR: LoadError: There was an error during testing
in expression starting at /cache/build/amdci7-2/julialang/julia-master/julia-609a4a0a1e/share/julia/stdlib/v1.8/Distributed/test/distributed_exec.jl:263
Distributed                         (1) |         failed at 2021-10-22T10:10:33.207

Full log: https://buildkite.com/julialang/julia-master/builds/5028#dfdf3a45-5510-402b-99d4-9daa860f0f09

[DilumAluthge]

I'm not sure if this change actually makes any sense.

[tkf]

If it's from multi-threaded tests, I don't think this is unexpected? I think it'd be better to move tester_linux64_mt (or only Distributed's tests) to experimental or skip this test when nthreads()> 1.

I think moving multithreaded tests for Distributed to experimental would be better than skipping it since we can do some post-hoc analysis on how much flaky it has been.

[DilumAluthge]

Hmmm. I definitely don't want to mark the entire tester_linux64_mt job as experimental, since all other test sets pass

I guess we could make a separate job that specifically runs the Distributed job with multithreading enabled, and mark that job as experimental.

[DilumAluthge]

I guess we could make a separate job that specifically runs the Distributed job with multithreading enabled, and mark that job as experimental.

Hmmm. This doesn't fix the case where someone runs the tests locally with e.g. JULIA_NUM_THREADS=16 ./julia test/runtests.jl all.

If this test is known/expected to sometimes fail when Julia was started with multiple threads, I think I would prefer to go with your suggestion of skipping those specific tests if Threads.nthreads > 1.

[DilumAluthge]

Actually, we can do both. We can skip it by default, but have a Buildkite job that doesn't skip it.

[DilumAluthge]

I'll do this in two parts:

The first part (skip these tests when Threads.nthreads() > 1, whether we are on CI or running locally) is here: #42764

I'll make the PR for the second part (experimental Buildkite job that runs the Distributed test suite with Threads.nthreads() > 1) once #42764 is merged.