You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/source/backends/json.rst
+35-2Lines changed: 35 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -92,7 +92,6 @@ propagate the exception thrown by Niels Lohmann's library.
92
92
93
93
The (keys) names ``"attributes"``, ``"data"`` and ``"datatype"`` are reserved and must not be used for base/mesh/particles path, records and their components.
94
94
95
-
A parallel (i.e. MPI) implementation is *not* available.
96
95
97
96
TOML Restrictions
98
97
-----------------
@@ -106,7 +105,41 @@ TOML does not support null values.
106
105
107
106
The (keys) names ``"attributes"``, ``"data"`` and ``"datatype"`` are reserved and must not be used for base/mesh/particles path, records and their components.
108
107
109
-
A parallel (i.e. MPI) implementation is *not* available.
108
+
109
+
Using in parallel (MPI)
110
+
-----------------------
111
+
112
+
Parallel I/O is not a first-class citizen in the JSON and TOML backends, and neither backend will "go out of its way" to support parallel workflows.
113
+
114
+
However there is a rudimentary form of read and write support in parallel:
115
+
116
+
Parallel reading
117
+
................
118
+
119
+
In order not to overload the parallel filesystem with parallel reads, read access to JSON datasets is done by rank 0 and then broadcast to all other ranks.
120
+
Note that there is no granularity whatsoever in reading a JSON file.
121
+
A JSON file is always read into memory and broadcast to all other ranks in its entirety.
122
+
123
+
Parallel writing
124
+
................
125
+
126
+
When executed in an MPI context, the JSON/TOML backends will not directly output a single text file, but instead a folder containing one file per MPI rank.
127
+
Neither backend will perform any data aggregation at all.
128
+
129
+
.. note::
130
+
131
+
The parallel write support of the JSON/TOML backends is intended mainly for debugging and prototyping workflows.
132
+
133
+
The folder will use the specified Series name, but append the postfix ``.parallel``.
134
+
(This is a deliberate indication that this folder cannot directly be opened again by the openPMD-api as a JSON/TOML dataset.)
135
+
This folder contains for each MPI rank *i* a file ``mpi_rank_<i>.json`` (resp. ``mpi_rank_<i>.toml``), containing the serial output of that rank.
136
+
A ``README.txt`` with basic usage instructions is also written.
137
+
138
+
.. note::
139
+
140
+
There is no direct support in the openPMD-api to read a JSON/TOML dataset written in this parallel fashion. The single files (e.g. ``data.json.parallel/mpi_rank_0.json``) are each valid openPMD files and can be read separately, however.
141
+
142
+
Note that the auxiliary function ``json::merge()`` (or in Python ``openpmd_api.merge_json()``) is not adequate for merging the single JSON/TOML files back into one, since it does not merge anything below the array level.
0 commit comments