ADIOS Restart: Avoid NxN access pattern with Aggregators

[ax3l]

During restart, the written N files by N aggregators (N <= |MPI_Size|)
are read again by each rank which causes NxN open files and fails for
large numbers of aggregators on parallel file systems.

We read our individual chunk now instead and perform a rather light
weight |MPI_Scan| on it to retreive the offsets.

Note 1: This bug is basically the same problem we had once with the
|libSplash| |SerialDataCollector| when we assumed that in general, a
re-sorting of GPUs /could/ have happened. (A pre-|alpha| bug during
GB2013.) On the ADIOS side, this can actually be achieved very
efficiently with an |ADIOS_READ_METHOD_BP_AGGREGATE| but is still very
experimental (did not work with zero-reads and/or zlib transport enabled).

Note 2: The HDF5 implementation

picongpu/src/picongpu/include/plugins/hdf5/restart/LoadSpecies.hpp

Lines 114 to 137 in f0ba515

	uint64Quint particlesInfo[gc.getGlobalSize()];
	Dimensions particlesInfoSizeRead;
	params->dataCollector->read(params->currentStep,
	(std::string(subGroup) + std::string("/particles_info")).c_str(),
	particlesInfoSizeRead,
	particlesInfo);
	assert(particlesInfoSizeRead[0] == gc.getGlobalSize());
	/* search my entry (using my scalar position) in particlesInfo */
	uint64_t particleOffset = 0;
	uint64_t myScalarPos = gc.getScalarPosition();
	for (size_t i = 0; i < particlesInfoSizeRead[0]; ++i)
	{
	if (particlesInfo[i][1] == myScalarPos)
	{
	totalNumParticles = particlesInfo[i][0];
	break;
	}
	particleOffset += particlesInfo[i][0];
	}

can be optimized the same way, nevertheless due to our MPI-I/O read
there it is not as severe as with aggregations right now.

• tested with large scale runs and lwfa moving window

[ax3l]

• 5

[psychocoderHPC]

the code changes looks good!

[ax3l]

remove comment

[ax3l]

weird, but it's not restarting...

runs fine until

[...]
PIConGPUVerbose MEMORY(8) | free mem after all particles are initialized 383 MiB
initialization time: 19sec 854msec = 19 sec
 50 % =      250 | time elapsed:                    0msec | avg time per step:   0msec

and then hangs.

Not all ranks reach

• "[...] used frames to load particles [...]" (wc -l is 60 instead of 2*48 = 96)
• Only 24 ranks reach "( end ) load species: i" and only 36 reach "( end ) load species: e"

Update: the last two messages only appear if the rank actually has >0 particles at all... -> will be updated for at least the ( end ) load species: output...

[ax3l]

Use MPI_Exscan for exclusive scan (without rank i):

Careful: The value in recvbuf on process 0 is undefined and unreliable as recvbuf is not significant for process 0. The value of recvbuf on process 1 is always the value in sendbuf on process 0.

[ax3l]

totalNumParticles is of uint64_cu and might not go well with uint64_t ...

[ax3l]

bug: uint32_t leftOverParticles should be uint64_t

[ax3l]

the copySpeciesGlobal2Local should also treat all arguments as 64bit...

[psychocoderHPC]

uint32_t should currently ok because the GridBuffer can only handle int (32Bit). In the most PMacc operations you can only allocate 2^31 elements (not byte).
But true since we use mallocMC we can create more than 2^31 particles per species.

[ax3l]

yes, the copySpeciesGlobal2Local seems valid after a check (for less then 2 billion particles per GPU).

[ax3l]

hm, not all ranks reach "load particles on device chunk offset=0" ...

[psychocoderHPC]

Yes, this is the current limit of PMacc to save a many regisers. I think we can change this for particles in the near feature.

[psychocoderHPC]

Can you check if all ranks leaf the adios call and enter MPI_scan?

[ax3l]

yes, 96 ranks reach "... particles from offset ... " the lines below

[ax3l]

hm, the "ADIOS: ( end ) load species" is in the "if" of ">0 particles"...

[psychocoderHPC]

FreeMemory is also in the same if condition but the allocation is outside.

Some posts before you wrote that not all ranks reach ... particles from offset .... Is this problem now solved?

[psychocoderHPC]

Ahh ok I can answer my last question by my self. The message is from the for loop were particles were loaded chunk wise and only ranks were something is to do enter this part of code.

[ax3l]

not sure, I am mixing uint64_cu and uint64_t in MPI_Scan - I will try that again, but all seem to leave the scan again...

[ax3l]

I have to declutter these if's so they also show up for ==0 particles loaded.

[ax3l]

This is for sanity. I am not sure if undefined means "guaranteed to be not overwritten" in this manual

 The value in recvbuf on process 0 is undefined and unreliable as recvbuf
is not significant for process 0.

[ax3l]

After the last commits, all 48 ranks reach for both species the ( end ) load species: output.

nevertheless, the whole sim still hangs after that outputting as before:

initialization time: 20sec 793msec = 20 sec
 50 % =      250 | time elapsed:                    0msec | avg time per step:   0msec

and that's it ^^

[ax3l]

would it be saver to use gc.getCommunicator().getMPIComm() here?
adiosComm is initialized in pluginLoad via a MPI_Comm_dup from it.
adios_read_init_method uses adiosComm afterward.

[ax3l]

in ADIOSCountParticles.hpp we use MPI_UNSIGNED_LONG_LONG for uint64_t values

[ax3l]

Hm, could it be possible that the ranks we use here are not the same as in gc.getScalarPosition() ? That would explain a lot ^^

[ax3l]

getScalarPosition() in libPMacc/include/mappings/simulation/GridController.hpp comes from

uint32_t getScalarPosition() const
            {
                return DataSpaceOperations<DIM>::map(getGpuNodes(), getPosition());
            }

hm... I would really like to use a MPI_Scan here, it scales (log N) way better than a crumpy MPI_Allgather ^^

[ax3l]

ah crap, that's a communicator and grid controller design flaw... I will use the less effcient Allgather now, but we should refactor the thing s othat getScalarPosition can actually be used with MPI communicators again 🔓

[ax3l]

@psychocoderHPC for the first time in my life, I found a useful application of MPI_Scan. We really need to redesign the communicator so we can use more efficient collectives...

P.S.: still not running, but due to an std::bad_alloc during attribute reads... too tired to continue just now, I must have mixed up some lengths ;)

[ax3l]

yes that's the plan. the question is if I can apply a fix to load the
current scheme for loads, so I do not have to change it twice.

[ax3l]

to be consistent, that would be the actual rank (but does not matter due to allgather).

[ax3l]

i*5

[ax3l]

just needs a moving window test, besides that it works on both small and very large scale runs.

[ax3l]

rebase me :) (#907 was merged)

[psychocoderHPC]

I get the flowing error with LWFA (enabled moving window)

[1,2]<stderr>:ERROR: Null pointer passed as file to adios_inq_var()
[1,2]<stderr>:ERROR: Null pointer passed as file to adios_perform_reads()
[1,2]<stderr>:ADIOS: error at cmd 'adios_perform_reads( params->fp, 1 )' (-4) in /bigdata/hplsim/scratch/widera/dev/src/picongpu/include/plugins/adios/restart/LoadParticleAttributesFromADIOS.hpp:113Null pointer passed as file to adios_perform_reads()

with debug output

...
[1,7]<stdout>:PIConGPUVerbose INPUT_OUTPUT(32) | ADIOS: Loading 0 particles from offset 140733876213206
[1,7]<stdout>:PIConGPUVerbose INPUT_OUTPUT(32) | ADIOS: malloc mapped memory: e

add additional own debug output to check the pointer address

...
[1,7]<stdout>:PIConGPUVerbose INPUT_OUTPUT(32) | ADIOS:  ptr before read /data/3000/particles/e/position/x from e: 0
...

[ax3l]

looks like failed open or a memory corruption. let's investigate.

[ax3l]

-> you are working on outdated code, pls checkout this branch (forgot to git fetch?)

[psychocoderHPC]

NOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
git fetch axel was missing

[psychocoderHPC]

OK all is fine.

LWFA (with moving window) is restart able and the pngs shows no difference.

please rebase this branch!

[ax3l]

cool, thx for testing! :)
rebased now to the latest version of our dev.