Solution invalidity sync failure

[lindsayad]

Bug Description

I just encountered the following failure when running the MOOSE test suite with -p 11

multiapps/sub_cycling.group/test: ===================================================================================
multiapps/sub_cycling.group/test: =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
multiapps/sub_cycling.group/test: =   PID 66470 RUNNING AT cone.hpc.inl.gov
multiapps/sub_cycling.group/test: =   EXIT CODE: 134
multiapps/sub_cycling.group/test: =   CLEANING UP REMAINING PROCESSES
multiapps/sub_cycling.group/test: =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
multiapps/sub_cycling.group/test: ===================================================================================
multiapps/sub_cycling.group/test: YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
multiapps/sub_cycling.group/test: This typically refers to a problem with your application.
multiapps/sub_cycling.group/test: Please see the FAQ page for debugging suggestions
multiapps/sub_cycling.group/test: [cone:66470:0:66470]       mm_ep.c:458  Assertion `ucs_arbiter_group_is_empty(&ep->arb_group)' failed
multiapps/sub_cycling.group/test: ==== backtrace (tid:  66470) ====
multiapps/sub_cycling.group/test:  0  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libucs.so.0(ucs_handle_error+0x75) [0x7fe2085b327d]
multiapps/sub_cycling.group/test:  1  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libucs.so.0(ucs_fatal_error_message+0xe7) [0x7fe2085b0519]
multiapps/sub_cycling.group/test:  2  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libucs.so.0(+0x3f6ef) [0x7fe2085b06ef]
multiapps/sub_cycling.group/test:  3  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libuct.so.0(uct_mm_ep_pending_add+0x85) [0x7fe21a260a0c]
multiapps/sub_cycling.group/test:  4  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libucp.so.0(ucp_request_pending_add+0x101) [0x7fe1c3c16e2a]
multiapps/sub_cycling.group/test:  5  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libucp.so.0(ucp_tag_send_nbx+0x226c) [0x7fe1c3d710f4]
multiapps/sub_cycling.group/test:  6  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x696ee2) [0x7fe1c46e4ee2]
multiapps/sub_cycling.group/test:  7  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x697548) [0x7fe1c46e5548]
multiapps/sub_cycling.group/test:  8  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x699468) [0x7fe1c46e7468]
multiapps/sub_cycling.group/test:  9  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x6995fe) [0x7fe1c46e75fe]
multiapps/sub_cycling.group/test: 10  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x69cafc) [0x7fe1c46eaafc]
multiapps/sub_cycling.group/test: 11  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x845ee0) [0x7fe1c4893ee0]
multiapps/sub_cycling.group/test: 12  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x8465b2) [0x7fe1c48945b2]
multiapps/sub_cycling.group/test: 13  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x84b21b) [0x7fe1c489921b]
multiapps/sub_cycling.group/test: 14  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x84b6a0) [0x7fe1c48996a0]
multiapps/sub_cycling.group/test: 15  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x7d8418) [0x7fe1c4826418]
multiapps/sub_cycling.group/test: 16  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x72d418) [0x7fe1c477b418]
multiapps/sub_cycling.group/test: 17  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x72e5a4) [0x7fe1c477c5a4]
multiapps/sub_cycling.group/test: 18  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x72f732) [0x7fe1c477d732]
multiapps/sub_cycling.group/test: 19  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x72ec81) [0x7fe1c477cc81]
multiapps/sub_cycling.group/test: 20  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x72f81f) [0x7fe1c477d81f]
multiapps/sub_cycling.group/test: 21  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(+0x3ed8ff) [0x7fe1c443b8ff]
multiapps/sub_cycling.group/test: 22  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libmpi.so.12(PMPI_Test+0x2f) [0x7fe1c443b9e7]
multiapps/sub_cycling.group/test: 23  /data/lindad/projects/mpich-dbg-petsc/scripts/../libmesh/installed/lib/libtimpi_dbg.so.11(_ZN5TIMPI7Request4testEv+0x50) [0x7fe2421dac3c]
multiapps/sub_cycling.group/test: 24  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(+0x92896b1) [0x7fe2538c46b1]
multiapps/sub_cycling.group/test: 25  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(+0x9283b46) [0x7fe2538beb46]
multiapps/sub_cycling.group/test: 26  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(+0x927c0e3) [0x7fe2538b70e3]
multiapps/sub_cycling.group/test: 27  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN18SolutionInvalidity13syncIterationEv+0x1a2) [0x7fe2538abf40]
multiapps/sub_cycling.group/test: 28  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN19NonlinearSystemBase23computeResidualInternalERKSt3setIjSt4lessIjESaIjEE+0x3ad3) [0x7fe2531c21e7]
multiapps/sub_cycling.group/test: 29  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN19NonlinearSystemBase19computeResidualTagsERKSt3setIjSt4lessIjESaIjEE+0xde4) [0x7fe2531b695e]
multiapps/sub_cycling.group/test: 30  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN13FEProblemBase19computeResidualTagsERKSt3setIjSt4lessIjESaIjEE+0x100d) [0x7fe252cd4c67]
multiapps/sub_cycling.group/test: 31  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN13FEProblemBase23computeResidualInternalERKN7libMesh13NumericVectorIdEERS2_RKSt3setIjSt4lessIjESaIjEE+0xc4d) [0x7fe252cd1dc5]
multiapps/sub_cycling.group/test: 32  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN13FEProblemBase15computeResidualERKN7libMesh13NumericVectorIdEERS2_j+0x3c7) [0x7fe252ccfde5]
multiapps/sub_cycling.group/test: 33  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN13FEProblemBase18computeResidualSysERN7libMesh23NonlinearImplicitSystemERKNS0_13NumericVectorIdEERS4_+0xbf1) [0x7fe252ccf4db]
multiapps/sub_cycling.group/test: 34  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN24ComputeFDResidualFunctor8residualERKN7libMesh13NumericVectorIdEERS2_RNS0_23NonlinearImplicitSystemE+0x5f) [0x7fe2531987ab]
multiapps/sub_cycling.group/test: 35  /data/lindad/projects/mpich-dbg-petsc/scripts/../libmesh/installed/lib/libmesh_dbg.so.0(libmesh_petsc_snes_mffd_residual+0x701) [0x7fe245ab0c70]
multiapps/sub_cycling.group/test: 36  /data/lindad/projects/mpich-dbg-petsc/scripts/../libmesh/installed/lib/libmesh_dbg.so.0(libmesh_petsc_snes_mffd_interface+0x16e) [0x7fe245ab1658]
multiapps/sub_cycling.group/test: 37  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libpetsc.so.3.024(+0x2452d7f) [0x7fe23a1d9d7f]
multiapps/sub_cycling.group/test: 38  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libpetsc.so.3.024(+0x2737d76) [0x7fe23a4bed76]
multiapps/sub_cycling.group/test: 39  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libpetsc.so.3.024(MatMult+0x1277) [0x7fe23a588798]
multiapps/sub_cycling.group/test: 40  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libpetsc.so.3.024(PCApplyBAorAB+0x409f) [0x7fe23c92c0de]
multiapps/sub_cycling.group/test: 41  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libpetsc.so.3.024(+0x415c641) [0x7fe23bee3641]
multiapps/sub_cycling.group/test: 42  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libpetsc.so.3.024(+0x415f508) [0x7fe23bee6508]
multiapps/sub_cycling.group/test: 43  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libpetsc.so.3.024(+0x416101a) [0x7fe23bee801a]
multiapps/sub_cycling.group/test: 44  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libpetsc.so.3.024(+0x42a315c) [0x7fe23c02a15c]
multiapps/sub_cycling.group/test: 45  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libpetsc.so.3.024(KSPSolve+0x746) [0x7fe23c02da7b]
multiapps/sub_cycling.group/test: 46  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libpetsc.so.3.024(+0x4c9daab) [0x7fe23ca24aab]
multiapps/sub_cycling.group/test: 47  /data/lindad/projects/mpich-dbg-petsc/petsc/arch-moose/lib/libpetsc.so.3.024(SNESSolve+0x4d96) [0x7fe23cbd090d]
multiapps/sub_cycling.group/test: 48  /data/lindad/projects/mpich-dbg-petsc/scripts/../libmesh/installed/lib/libmesh_dbg.so.0(_ZN7libMesh20PetscNonlinearSolverIdE5solveERNS_12SparseMatrixIdEERNS_13NumericVectorIdEES7_dj+0x1abd) [0x7fe245abae07]
multiapps/sub_cycling.group/test: 49  /data/lindad/projects/mpich-dbg-petsc/scripts/../libmesh/installed/lib/libmesh_dbg.so.0(_ZN7libMesh23NonlinearImplicitSystem5solveEv+0x1f6) [0x7fe245b7e80c]
multiapps/sub_cycling.group/test: 50  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN15NonlinearSystem5solveEv+0xda7) [0x7fe2531ad771]
multiapps/sub_cycling.group/test: 51  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN13FEProblemBase5solveEj+0x9d1) [0x7fe252cc465d]
multiapps/sub_cycling.group/test: 52  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN14FEProblemSolve5solveEv+0x139) [0x7fe25184a3bb]
multiapps/sub_cycling.group/test: 53  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN15FixedPointSolve9solveStepERKSt3setImSt4lessImESaImEE+0x746) [0x7fe25184f852]
multiapps/sub_cycling.group/test: 54  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN15FixedPointSolve5solveEv+0xf47) [0x7fe25184e703]
multiapps/sub_cycling.group/test: 55  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN11TimeStepper4stepEv+0x46) [0x7fe2533fba9e]
multiapps/sub_cycling.group/test: 56  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN13TransientBase8takeStepEd+0x493) [0x7fe251860fd1]
multiapps/sub_cycling.group/test: 57  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN17TransientMultiApp9solveStepEddb+0xf93) [0x7fe252541219]
multiapps/sub_cycling.group/test: 58  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN13FEProblemBase13execMultiAppsE13MooseEnumItemb+0xb59) [0x7fe252cb8443]
multiapps/sub_cycling.group/test: 59  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN15FixedPointSolve9solveStepERKSt3setImSt4lessImESaImEE+0xab8) [0x7fe25184fbc4]
multiapps/sub_cycling.group/test: 60  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN15FixedPointSolve5solveEv+0xf47) [0x7fe25184e703]
multiapps/sub_cycling.group/test: 61  /data/lindad/projects/mpich-dbg-petsc/framework/libmoose-dbg.so.0(_ZN11TimeStepper4stepEv+0x46) [0x7fe2533fba9e]

Steps to Reproduce

I don't know if it will be regularly reproducible but my reproducing command is given above

Impact

Brittle test harness

[Optional] Diagnostics

No response

[lindsayad]

FYI @MengnanLi91 @GiudGiud

[GiudGiud]

We do MPI calls to synch the counts across ranks on every iteration. Nearly every single test in the test suite does this btw.

I can't reproduce this locally on my M4. Can you reproduce this consistently on a platform?

[lindsayad]

No. It's possible it only happens under heavy system load