Passing of advanced backend parameters

[anokfireball]

An option to provide advanced parameters specific to a backend is desirable. This can include MPI_Info, file access strategies,communication strategies, buffer sizes among other things. This should be optional with sane defaults for all values.

Specifically, this matters in two places:

• Series, where one would provide general backend parameters shared across all files, paths, datasets and attributes in the Series (e.g. MPI_Info, ADIOS_AGGREGATE, nometa)

namespace openPMD
{
Series::Series(std::string const& filepath,
              AccessType at,
              MPI_Comm comm,
              AbstractSeriesParam* = nullptr /* default */);
} // openPMD

• Datasets, where one would provide parameters different for specific datasets (e.g. compression/transform)

namespace openPMD
{
Dataset&
Dataset::setParameters(AbstractDatasetParam*);
} // openPMD

Implementation can be done via polymorphic parameter structs (similar to IO::Parameter) where paramters are passed named and typesafe (preferred) or as a map (similar to backend::Attributable) (not preferred).

[ax3l]

I would suggest to pass an object by value for AbstractSeriesParam. One can still add a default dummy as default arg:

Series::Series(std::string const& filepath,
              AccessType at,
              MPI_Comm comm,
              AbstractSeriesParam = SeriesDefaultParam{});

We can also add a Series::setOptions() method for later adjustment, but some flags might already be needed at construction time.

[franzpoeschel]

To get back on this, a further option could be using JSON. We already ship with Niels Lohmann's JSON library. This would allow to flexibly express backend-specific configuration options while at the same time not requiring to implement backend-specific classes like the solution with AbstractSeriesParam.

As an example, choosing the ADIOS2 engine with a JSON configuration:

{
  "Engine": {
    "Type": "SST",
    "Params": {
      "QueueLimit": 3,
      "QueueFullPolicy": "Block"
    }
  }
}

This can be easily exposed to the Python API by passing the JSON configuration as a simple string.

[ax3l]

Regarding HDF5 options and #554 : we can also try to expose H5FD_MPIO_INDEPENDENT again as @RemiLehe just mentioned offline. We did this in the past in libSplash but had no success with it, e.g. deadlocks or very bad parallel performance. We can see if MPI-IO implementations improved by just exposing it.

[franzpoeschel]

I have started work on backend-specific configuration options, using Niels Lohmann's JSON library (which is already in usage for the openPMD API's JSON backend).
I will publish and link this after cleaning things up for a bit.

[ax3l]

Thanks, just ping us on a WIP PR? :)

Another thing I want to add as an option: in case we build with ADIOS1 and ADIOS2 support, it would be great to choose the backend priority at runtime (for e.g. benchmarks) instead of having to re-compile.