Skip to content

Commit a564618

Browse files
committed
Add documentation
1 parent 789ba98 commit a564618

File tree

1 file changed

+35
-2
lines changed

1 file changed

+35
-2
lines changed

docs/source/backends/json.rst

Lines changed: 35 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,6 @@ propagate the exception thrown by Niels Lohmann's library.
9292

9393
The (keys) names ``"attributes"``, ``"data"`` and ``"datatype"`` are reserved and must not be used for base/mesh/particles path, records and their components.
9494

95-
A parallel (i.e. MPI) implementation is *not* available.
9695

9796
TOML Restrictions
9897
-----------------
@@ -106,7 +105,41 @@ TOML does not support null values.
106105

107106
The (keys) names ``"attributes"``, ``"data"`` and ``"datatype"`` are reserved and must not be used for base/mesh/particles path, records and their components.
108107

109-
A parallel (i.e. MPI) implementation is *not* available.
108+
109+
Using in parallel (MPI)
110+
-----------------------
111+
112+
Parallel I/O is not a first-class citizen in the JSON and TOML backends, and neither backend will "go out of its way" to support parallel workflows.
113+
114+
However there is a rudimentary form of read and write support in parallel:
115+
116+
Parallel reading
117+
................
118+
119+
In order not to overload the parallel filesystem with parallel reads, read access to JSON datasets is done by rank 0 and then broadcast to all other ranks.
120+
Note that there is no granularity whatsoever in reading a JSON file.
121+
A JSON file is always read into memory and broadcast to all other ranks in its entirety.
122+
123+
Parallel writing
124+
................
125+
126+
When executed in an MPI context, the JSON/TOML backends will not directly output a single text file, but instead a folder containing one file per MPI rank.
127+
Neither backend will perform any data aggregation at all.
128+
129+
.. note::
130+
131+
The parallel write support of the JSON/TOML backends is intended mainly for debugging and prototyping workflows.
132+
133+
The folder will use the specified Series name, but append the postfix ``.parallel``.
134+
(This is a deliberate indication that this folder cannot directly be opened again by the openPMD-api as a JSON/TOML dataset.)
135+
This folder contains for each MPI rank *i* a file ``mpi_rank_<i>.json`` (resp. ``mpi_rank_<i>.toml``), containing the serial output of that rank.
136+
A ``README.txt`` with basic usage instructions is also written.
137+
138+
.. note::
139+
140+
There is no direct support in the openPMD-api to read a JSON/TOML dataset written in this parallel fashion. The single files (e.g. ``data.json.parallel/mpi_rank_0.json``) are each valid openPMD files and can be read separately, however.
141+
142+
Note that the auxiliary function ``json::merge()`` (or in Python ``openpmd_api.merge_json()``) is not adequate for merging the single JSON/TOML files back into one, since it does not merge anything below the array level.
110143

111144

112145
Example

0 commit comments

Comments
 (0)