Skip to content

Conversation

@franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented Jun 5, 2019

Issue The fact that the class ParticleSpecies will enforce openPMD requirements currently leads to the awkward situation that those records have to be defined using RecordComponent::resetDataset( … ) before issuing the first flush on the Series object. A later RecordComponent::storeChunk( … ) is fine, but the datasets need to be defined up front.
This is (1) an awkward pitfall, (2) leads to rather confusing errors if done wrongly (what(): A Record can not be written without any contained RecordComponents: position) and (3) leads to ugly workarounds in workflows where the single records of a particle species are defined and written one after another (e.g. the openPMD plugin for picongpu that I am currently working on).

Suggested Change Move these two lines into the flush method override of ParticleSpecies, but do not create the records if not present.

Open questions The above solution will ensure openPMD requirements for position[Offset] records if the user remembers to create them. I have not (yet) implemented an alternative check if the user forgets to create those records. The logical location for such a check would be the destructor, but throwing exceptions in destructors is discouraged.

First commit adds new tests (and removes some existing ones), second commit will implement the suggested change.

franzpoeschel added a commit to franzpoeschel/picongpu that referenced this pull request Jun 5, 2019
Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes.
The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518.
This will ensure openPMD specific requirements for those records at flush time as soon as those records are present -- rather than at creation time of the particle species object.
Currently missing is any kind of check that those records are present before closing the series.
@ax3l
Copy link
Member

ax3l commented Jun 6, 2019

Thank you, great this gets addressed.

I think in general that we should remove the creation logic from the objects and divide "openPMD objects" from "required and default attribute creation" in our implementation. This is too interwinded still and causes issues.

Furthermore, I think we should relax what happens on .flush() to purely flush buffers and check and create default structures only on Series close (destruction) or any additional member function that we introduce for that, e.g. .complete()/.close() (e.g. also on iterations!) or so. cc @C0nsultant

The logical location for such a check would be the destructor, but throwing exceptions in destructors is discouraged.

Good point. Maybe we just need an explicit .close() on Series and even Iteration objects. #149

@ax3l
Copy link
Member

ax3l commented Jun 6, 2019

Before we merge this, did you verify the created files still pass the validator? https://github.com/openPMD/openPMD-validator

@franzpoeschel
Copy link
Contributor Author

Did that just now, the files yield exactly the same errors as those on the dev branch do, so I guess there's further things to investigate. I checked the JSON files with diff now and there are actually differences for the serial_patch and constant_scalar tests where the unit dimensions are erroneously all zero, so I'll need to check that.

@ax3l
Copy link
Member

ax3l commented Jun 6, 2019

Ok, let's do that in a follow-up PR.

note: unitDimensions can be 7x zero, e.g. for unitless counts.

@ax3l ax3l merged commit c2df66a into openPMD:dev Jun 6, 2019
franzpoeschel added a commit to franzpoeschel/picongpu that referenced this pull request Jun 13, 2019
Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes.
The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518.
franzpoeschel added a commit to franzpoeschel/picongpu that referenced this pull request Mar 4, 2020
Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes.
The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518.
franzpoeschel added a commit to franzpoeschel/picongpu that referenced this pull request Mar 25, 2020
Create openPMDWriter based on ADIOSWriter (WIP)

Adapt CountParticles partially to openPMD

Adapt WriteSpecies to openPMD

Translate WriteMeta.hpp to openPMD

Add function "asStandardVector()"

Translate NDScalars.hpp to openPMD

Adapt everything to openPMD except for the main file (openPMDWriter.hpp)

Adapt openPMDWriter.hpp to openPMD WIP

Change management of openPMD Series

Further adapt openPMDWriter to openPMD

Add openPMD to CMake

Add openPMD to version formatting

Properly acquaint openPMD plugin with build and help system

Make openPMD namespacing explicit

Remove adios-specific fields from ThreadParams

First successfull run with LaserWakeField example

Cleanup

Use clang-format

Update licensing information

Separate basename and filename extension into two separate help parameters

Refactor dataset creation

Use Mesh::setTimeOffset() template

Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch.

Clean up some leftovers

Use ASCII characters to spell my name

Remove unnecessary whitespaces

Adapt to removal of pmacc::forward

Remove accidentally versioned config file

Make checkpoints usable

Fix a number of bugs concerning the reading of checkpoints
Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation.

Remove CollectFieldsSizes

Legacy from ADIOS writer, not necessary in openPMD.

Remove ParticleAttributeSize and openPMDCountParticles

Legacy from ADIOS Writer

Use clang-format

Adhere to openPMD requirements before flushing

For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush.

Maybe open an issue in the openPMD API to allow for a more flexible usage.

Fix an indexing bug

Eliminate dataset preparation for NDScalars

Also fix a bug where particles were named wrongly in checkpoints.

Do not write empty particle datasets

Treat non-existing datasets as empty in reading

Remove prepared datasets

Remove WithWindow struct

Use transform to enable ADIOS1 compression

Remove accidentally versioned files

Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD

Remove traces of the old ADIOS plugin

mostly the word "adios" from comments

Take copies and not references of openPMD handles

Fix autoformatting

Require newer openPMD API

Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH

Add version label to format string only if present

Replace typedefs with using

Remove further indexing bug

in writing particles_info table

Cleanup restart

Remove dataset preparation

Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes.
The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518.

Postpone writing of meta attributes

Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles.
Bug report will follow.

Resolve counting bug during particle writing

Fix whitespaces

Separate ADIOS and HDF5 particle writing strategies

Allow choosing strategy as runtime parameter

Cleanup

Fix openPMD version formatting

Update examples to use openPMD

Refactor passing of filename and extension

Reassemble filename and extension upon opening the series.

Fix some missing includes

Do not skip writing empty datasets

See openPMD PR: openPMD/openPMD-api#529
This allows to write empty datasets

Remove debugging leftovers

Write timeOffset for particles in a standard-compliant way

Do not declare zero-sized arrays

C++ standard requires that array size evaluate to a value greater than zero.

Do not use storeChunk on empty datasets

centralize initialization of thread params from config

Fix undefined identifier in assert statements

Error passes silently in release builds.

Pass JSON config to openPMD::Series ctor

Do not copy Series object

see openPMD/openPMD-api#534

Allow NULL as configuration infix to denote empty string

Adapt to changes in pmacc etc.

Enable use of group-based layout as well

Requires keeping openPMD Series open. Since openPMD currently has no
explicit call to close a file, we implement this only for group-based
layout for now.

Do not use deprecated Series::setSoftwareVersion call

Apply commit b797c0a to openPMD backend

Formatting in .cfg files

Fix an uninitialized value and an indexing bug

Implement reviewers' comments concerning CMakeLists
franzpoeschel added a commit to franzpoeschel/picongpu that referenced this pull request Mar 25, 2020
Create openPMDWriter based on ADIOSWriter (WIP)

Adapt CountParticles partially to openPMD

Adapt WriteSpecies to openPMD

Translate WriteMeta.hpp to openPMD

Add function "asStandardVector()"

Translate NDScalars.hpp to openPMD

Adapt everything to openPMD except for the main file (openPMDWriter.hpp)

Adapt openPMDWriter.hpp to openPMD WIP

Change management of openPMD Series

Further adapt openPMDWriter to openPMD

Add openPMD to CMake

Add openPMD to version formatting

Properly acquaint openPMD plugin with build and help system

Make openPMD namespacing explicit

Remove adios-specific fields from ThreadParams

First successfull run with LaserWakeField example

Cleanup

Use clang-format

Update licensing information

Separate basename and filename extension into two separate help parameters

Refactor dataset creation

Use Mesh::setTimeOffset() template

Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch.

Clean up some leftovers

Use ASCII characters to spell my name

Remove unnecessary whitespaces

Adapt to removal of pmacc::forward

Remove accidentally versioned config file

Make checkpoints usable

Fix a number of bugs concerning the reading of checkpoints
Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation.

Remove CollectFieldsSizes

Legacy from ADIOS writer, not necessary in openPMD.

Remove ParticleAttributeSize and openPMDCountParticles

Legacy from ADIOS Writer

Use clang-format

Adhere to openPMD requirements before flushing

For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush.

Maybe open an issue in the openPMD API to allow for a more flexible usage.

Fix an indexing bug

Eliminate dataset preparation for NDScalars

Also fix a bug where particles were named wrongly in checkpoints.

Do not write empty particle datasets

Treat non-existing datasets as empty in reading

Remove prepared datasets

Remove WithWindow struct

Use transform to enable ADIOS1 compression

Remove accidentally versioned files

Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD

Remove traces of the old ADIOS plugin

mostly the word "adios" from comments

Take copies and not references of openPMD handles

Fix autoformatting

Require newer openPMD API

Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH

Add version label to format string only if present

Replace typedefs with using

Remove further indexing bug

in writing particles_info table

Cleanup restart

Remove dataset preparation

Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes.
The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518.

Postpone writing of meta attributes

Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles.
Bug report will follow.

Resolve counting bug during particle writing

Fix whitespaces

Separate ADIOS and HDF5 particle writing strategies

Allow choosing strategy as runtime parameter

Cleanup

Fix openPMD version formatting

Update examples to use openPMD

Refactor passing of filename and extension

Reassemble filename and extension upon opening the series.

Fix some missing includes

Do not skip writing empty datasets

See openPMD PR: openPMD/openPMD-api#529
This allows to write empty datasets

Remove debugging leftovers

Write timeOffset for particles in a standard-compliant way

Do not declare zero-sized arrays

C++ standard requires that array size evaluate to a value greater than zero.

Do not use storeChunk on empty datasets

centralize initialization of thread params from config

Fix undefined identifier in assert statements

Error passes silently in release builds.

Pass JSON config to openPMD::Series ctor

Do not copy Series object

see openPMD/openPMD-api#534

Allow NULL as configuration infix to denote empty string

Adapt to changes in pmacc etc.

Enable use of group-based layout as well

Requires keeping openPMD Series open. Since openPMD currently has no
explicit call to close a file, we implement this only for group-based
layout for now.

Do not use deprecated Series::setSoftwareVersion call

Apply commit b797c0a to openPMD backend

Formatting in .cfg files

Fix an uninitialized value and an indexing bug

Implement reviewers' comments concerning CMakeLists
franzpoeschel added a commit to franzpoeschel/picongpu that referenced this pull request Mar 26, 2020
Create openPMDWriter based on ADIOSWriter (WIP)

Adapt CountParticles partially to openPMD

Adapt WriteSpecies to openPMD

Translate WriteMeta.hpp to openPMD

Add function "asStandardVector()"

Translate NDScalars.hpp to openPMD

Adapt everything to openPMD except for the main file (openPMDWriter.hpp)

Adapt openPMDWriter.hpp to openPMD WIP

Change management of openPMD Series

Further adapt openPMDWriter to openPMD

Add openPMD to CMake

Add openPMD to version formatting

Properly acquaint openPMD plugin with build and help system

Make openPMD namespacing explicit

Remove adios-specific fields from ThreadParams

First successfull run with LaserWakeField example

Cleanup

Use clang-format

Update licensing information

Separate basename and filename extension into two separate help parameters

Refactor dataset creation

Use Mesh::setTimeOffset() template

Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch.

Clean up some leftovers

Use ASCII characters to spell my name

Remove unnecessary whitespaces

Adapt to removal of pmacc::forward

Remove accidentally versioned config file

Make checkpoints usable

Fix a number of bugs concerning the reading of checkpoints
Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation.

Remove CollectFieldsSizes

Legacy from ADIOS writer, not necessary in openPMD.

Remove ParticleAttributeSize and openPMDCountParticles

Legacy from ADIOS Writer

Use clang-format

Adhere to openPMD requirements before flushing

For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush.

Maybe open an issue in the openPMD API to allow for a more flexible usage.

Fix an indexing bug

Eliminate dataset preparation for NDScalars

Also fix a bug where particles were named wrongly in checkpoints.

Do not write empty particle datasets

Treat non-existing datasets as empty in reading

Remove prepared datasets

Remove WithWindow struct

Use transform to enable ADIOS1 compression

Remove accidentally versioned files

Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD

Remove traces of the old ADIOS plugin

mostly the word "adios" from comments

Take copies and not references of openPMD handles

Fix autoformatting

Require newer openPMD API

Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH

Add version label to format string only if present

Replace typedefs with using

Remove further indexing bug

in writing particles_info table

Cleanup restart

Remove dataset preparation

Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes.
The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518.

Postpone writing of meta attributes

Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles.
Bug report will follow.

Resolve counting bug during particle writing

Fix whitespaces

Separate ADIOS and HDF5 particle writing strategies

Allow choosing strategy as runtime parameter

Cleanup

Fix openPMD version formatting

Update examples to use openPMD

Refactor passing of filename and extension

Reassemble filename and extension upon opening the series.

Fix some missing includes

Do not skip writing empty datasets

See openPMD PR: openPMD/openPMD-api#529
This allows to write empty datasets

Remove debugging leftovers

Write timeOffset for particles in a standard-compliant way

Do not declare zero-sized arrays

C++ standard requires that array size evaluate to a value greater than zero.

Do not use storeChunk on empty datasets

centralize initialization of thread params from config

Fix undefined identifier in assert statements

Error passes silently in release builds.

Pass JSON config to openPMD::Series ctor

Do not copy Series object

see openPMD/openPMD-api#534

Allow NULL as configuration infix to denote empty string

Adapt to changes in pmacc etc.

Enable use of group-based layout as well

Requires keeping openPMD Series open. Since openPMD currently has no
explicit call to close a file, we implement this only for group-based
layout for now.

Do not use deprecated Series::setSoftwareVersion call

Apply commit b797c0a to openPMD backend

Formatting in .cfg files

Fix an uninitialized value and an indexing bug

Implement reviewers' comments concerning CMakeLists
franzpoeschel added a commit to franzpoeschel/picongpu that referenced this pull request Apr 4, 2020
Create openPMDWriter based on ADIOSWriter (WIP)

Adapt CountParticles partially to openPMD

Adapt WriteSpecies to openPMD

Translate WriteMeta.hpp to openPMD

Add function "asStandardVector()"

Translate NDScalars.hpp to openPMD

Adapt everything to openPMD except for the main file (openPMDWriter.hpp)

Adapt openPMDWriter.hpp to openPMD WIP

Change management of openPMD Series

Further adapt openPMDWriter to openPMD

Add openPMD to CMake

Add openPMD to version formatting

Properly acquaint openPMD plugin with build and help system

Make openPMD namespacing explicit

Remove adios-specific fields from ThreadParams

First successfull run with LaserWakeField example

Cleanup

Use clang-format

Update licensing information

Separate basename and filename extension into two separate help parameters

Refactor dataset creation

Use Mesh::setTimeOffset() template

Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch.

Clean up some leftovers

Use ASCII characters to spell my name

Remove unnecessary whitespaces

Adapt to removal of pmacc::forward

Remove accidentally versioned config file

Make checkpoints usable

Fix a number of bugs concerning the reading of checkpoints
Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation.

Remove CollectFieldsSizes

Legacy from ADIOS writer, not necessary in openPMD.

Remove ParticleAttributeSize and openPMDCountParticles

Legacy from ADIOS Writer

Use clang-format

Adhere to openPMD requirements before flushing

For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush.

Maybe open an issue in the openPMD API to allow for a more flexible usage.

Fix an indexing bug

Eliminate dataset preparation for NDScalars

Also fix a bug where particles were named wrongly in checkpoints.

Do not write empty particle datasets

Treat non-existing datasets as empty in reading

Remove prepared datasets

Remove WithWindow struct

Use transform to enable ADIOS1 compression

Remove accidentally versioned files

Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD

Remove traces of the old ADIOS plugin

mostly the word "adios" from comments

Take copies and not references of openPMD handles

Fix autoformatting

Require newer openPMD API

Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH

Add version label to format string only if present

Replace typedefs with using

Remove further indexing bug

in writing particles_info table

Cleanup restart

Remove dataset preparation

Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes.
The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518.

Postpone writing of meta attributes

Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles.
Bug report will follow.

Resolve counting bug during particle writing

Fix whitespaces

Separate ADIOS and HDF5 particle writing strategies

Allow choosing strategy as runtime parameter

Cleanup

Fix openPMD version formatting

Update examples to use openPMD

Refactor passing of filename and extension

Reassemble filename and extension upon opening the series.

Fix some missing includes

Do not skip writing empty datasets

See openPMD PR: openPMD/openPMD-api#529
This allows to write empty datasets

Remove debugging leftovers

Write timeOffset for particles in a standard-compliant way

Do not declare zero-sized arrays

C++ standard requires that array size evaluate to a value greater than zero.

Do not use storeChunk on empty datasets

centralize initialization of thread params from config

Fix undefined identifier in assert statements

Error passes silently in release builds.

Pass JSON config to openPMD::Series ctor

Do not copy Series object

see openPMD/openPMD-api#534

Allow NULL as configuration infix to denote empty string

Adapt to changes in pmacc etc.

Enable use of group-based layout as well

Requires keeping openPMD Series open. Since openPMD currently has no
explicit call to close a file, we implement this only for group-based
layout for now.

Do not use deprecated Series::setSoftwareVersion call

Apply commit b797c0a to openPMD backend

Formatting in .cfg files

Fix an uninitialized value and an indexing bug

Implement reviewers' comments concerning CMakeLists
franzpoeschel added a commit to franzpoeschel/picongpu that referenced this pull request Apr 25, 2020
Create openPMDWriter based on ADIOSWriter (WIP)

Adapt CountParticles partially to openPMD

Adapt WriteSpecies to openPMD

Translate WriteMeta.hpp to openPMD

Add function "asStandardVector()"

Translate NDScalars.hpp to openPMD

Adapt everything to openPMD except for the main file (openPMDWriter.hpp)

Adapt openPMDWriter.hpp to openPMD WIP

Change management of openPMD Series

Further adapt openPMDWriter to openPMD

Add openPMD to CMake

Add openPMD to version formatting

Properly acquaint openPMD plugin with build and help system

Make openPMD namespacing explicit

Remove adios-specific fields from ThreadParams

First successfull run with LaserWakeField example

Cleanup

Use clang-format

Update licensing information

Separate basename and filename extension into two separate help parameters

Refactor dataset creation

Use Mesh::setTimeOffset() template

Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch.

Clean up some leftovers

Use ASCII characters to spell my name

Remove unnecessary whitespaces

Adapt to removal of pmacc::forward

Remove accidentally versioned config file

Make checkpoints usable

Fix a number of bugs concerning the reading of checkpoints
Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation.

Remove CollectFieldsSizes

Legacy from ADIOS writer, not necessary in openPMD.

Remove ParticleAttributeSize and openPMDCountParticles

Legacy from ADIOS Writer

Use clang-format

Adhere to openPMD requirements before flushing

For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush.

Maybe open an issue in the openPMD API to allow for a more flexible usage.

Fix an indexing bug

Eliminate dataset preparation for NDScalars

Also fix a bug where particles were named wrongly in checkpoints.

Do not write empty particle datasets

Treat non-existing datasets as empty in reading

Remove prepared datasets

Remove WithWindow struct

Use transform to enable ADIOS1 compression

Remove accidentally versioned files

Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD

Remove traces of the old ADIOS plugin

mostly the word "adios" from comments

Take copies and not references of openPMD handles

Fix autoformatting

Require newer openPMD API

Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH

Add version label to format string only if present

Replace typedefs with using

Remove further indexing bug

in writing particles_info table

Cleanup restart

Remove dataset preparation

Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes.
The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518.

Postpone writing of meta attributes

Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles.
Bug report will follow.

Resolve counting bug during particle writing

Fix whitespaces

Separate ADIOS and HDF5 particle writing strategies

Allow choosing strategy as runtime parameter

Cleanup

Fix openPMD version formatting

Update examples to use openPMD

Refactor passing of filename and extension

Reassemble filename and extension upon opening the series.

Fix some missing includes

Do not skip writing empty datasets

See openPMD PR: openPMD/openPMD-api#529
This allows to write empty datasets

Remove debugging leftovers

Write timeOffset for particles in a standard-compliant way

Do not declare zero-sized arrays

C++ standard requires that array size evaluate to a value greater than zero.

Do not use storeChunk on empty datasets

centralize initialization of thread params from config

Fix undefined identifier in assert statements

Error passes silently in release builds.

Pass JSON config to openPMD::Series ctor

Do not copy Series object

see openPMD/openPMD-api#534

Allow NULL as configuration infix to denote empty string

Adapt to changes in pmacc etc.

Enable use of group-based layout as well

Requires keeping openPMD Series open. Since openPMD currently has no
explicit call to close a file, we implement this only for group-based
layout for now.

Do not use deprecated Series::setSoftwareVersion call

Apply commit b797c0a to openPMD backend

Formatting in .cfg files

Fix an uninitialized value and an indexing bug

Implement reviewers' comments concerning CMakeLists
franzpoeschel added a commit to franzpoeschel/picongpu that referenced this pull request May 6, 2020
Create openPMDWriter based on ADIOSWriter (WIP)

Adapt CountParticles partially to openPMD

Adapt WriteSpecies to openPMD

Translate WriteMeta.hpp to openPMD

Add function "asStandardVector()"

Translate NDScalars.hpp to openPMD

Adapt everything to openPMD except for the main file (openPMDWriter.hpp)

Adapt openPMDWriter.hpp to openPMD WIP

Change management of openPMD Series

Further adapt openPMDWriter to openPMD

Add openPMD to CMake

Add openPMD to version formatting

Properly acquaint openPMD plugin with build and help system

Make openPMD namespacing explicit

Remove adios-specific fields from ThreadParams

First successfull run with LaserWakeField example

Cleanup

Use clang-format

Update licensing information

Separate basename and filename extension into two separate help parameters

Refactor dataset creation

Use Mesh::setTimeOffset() template

Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch.

Clean up some leftovers

Use ASCII characters to spell my name

Remove unnecessary whitespaces

Adapt to removal of pmacc::forward

Remove accidentally versioned config file

Make checkpoints usable

Fix a number of bugs concerning the reading of checkpoints
Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation.

Remove CollectFieldsSizes

Legacy from ADIOS writer, not necessary in openPMD.

Remove ParticleAttributeSize and openPMDCountParticles

Legacy from ADIOS Writer

Use clang-format

Adhere to openPMD requirements before flushing

For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush.

Maybe open an issue in the openPMD API to allow for a more flexible usage.

Fix an indexing bug

Eliminate dataset preparation for NDScalars

Also fix a bug where particles were named wrongly in checkpoints.

Do not write empty particle datasets

Treat non-existing datasets as empty in reading

Remove prepared datasets

Remove WithWindow struct

Use transform to enable ADIOS1 compression

Remove accidentally versioned files

Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD

Remove traces of the old ADIOS plugin

mostly the word "adios" from comments

Take copies and not references of openPMD handles

Fix autoformatting

Require newer openPMD API

Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH

Add version label to format string only if present

Replace typedefs with using

Remove further indexing bug

in writing particles_info table

Cleanup restart

Remove dataset preparation

Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes.
The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518.

Postpone writing of meta attributes

Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles.
Bug report will follow.

Resolve counting bug during particle writing

Fix whitespaces

Separate ADIOS and HDF5 particle writing strategies

Allow choosing strategy as runtime parameter

Cleanup

Fix openPMD version formatting

Update examples to use openPMD

Refactor passing of filename and extension

Reassemble filename and extension upon opening the series.

Fix some missing includes

Do not skip writing empty datasets

See openPMD PR: openPMD/openPMD-api#529
This allows to write empty datasets

Remove debugging leftovers

Write timeOffset for particles in a standard-compliant way

Do not declare zero-sized arrays

C++ standard requires that array size evaluate to a value greater than zero.

Do not use storeChunk on empty datasets

centralize initialization of thread params from config

Fix undefined identifier in assert statements

Error passes silently in release builds.

Pass JSON config to openPMD::Series ctor

Do not copy Series object

see openPMD/openPMD-api#534

Allow NULL as configuration infix to denote empty string

Adapt to changes in pmacc etc.

Enable use of group-based layout as well

Requires keeping openPMD Series open. Since openPMD currently has no
explicit call to close a file, we implement this only for group-based
layout for now.

Do not use deprecated Series::setSoftwareVersion call

Apply commit b797c0a to openPMD backend

Formatting in .cfg files

Fix an uninitialized value and an indexing bug

Implement reviewers' comments concerning CMakeLists

Fix some bugs and remove unneeded parameters

Some ADIOS-specific parameters have been remove that haven't been
implemented anyway and are going to be implemented via JSON.

Don't set compression if set to "none".

Also, I missed a small part when porting b797c0a to this plugin.

Add documentation for openPMD plugin [WIP]

Further write documentation

Rename data preparation strategies

adios -> doubleBuffer
hdf5  -> mappedMemory
Old names may still be used alternatively.

Fix indexing bug for non domain-bound fields

Fix indexing order in fields

Implement Reviewer's comments

Remove workaround for ADIOS1 backend

fix attributes

Mesh record component positions and unitSI
Iteration: time
ParticleAttribute unitDimension

Implement reviewer's comments (2)

Update copyright headers of files changed

Add compression to openPMD TBG example

Use openPMD::getVersion

remove unnecessary parentheses
franzpoeschel added a commit to franzpoeschel/picongpu that referenced this pull request May 18, 2020
Create openPMDWriter based on ADIOSWriter (WIP)

Adapt CountParticles partially to openPMD

Adapt WriteSpecies to openPMD

Translate WriteMeta.hpp to openPMD

Add function "asStandardVector()"

Translate NDScalars.hpp to openPMD

Adapt everything to openPMD except for the main file (openPMDWriter.hpp)

Adapt openPMDWriter.hpp to openPMD WIP

Change management of openPMD Series

Further adapt openPMDWriter to openPMD

Add openPMD to CMake

Add openPMD to version formatting

Properly acquaint openPMD plugin with build and help system

Make openPMD namespacing explicit

Remove adios-specific fields from ThreadParams

First successfull run with LaserWakeField example

Cleanup

Use clang-format

Update licensing information

Separate basename and filename extension into two separate help parameters

Refactor dataset creation

Use Mesh::setTimeOffset() template

Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch.

Clean up some leftovers

Use ASCII characters to spell my name

Remove unnecessary whitespaces

Adapt to removal of pmacc::forward

Remove accidentally versioned config file

Make checkpoints usable

Fix a number of bugs concerning the reading of checkpoints
Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation.

Remove CollectFieldsSizes

Legacy from ADIOS writer, not necessary in openPMD.

Remove ParticleAttributeSize and openPMDCountParticles

Legacy from ADIOS Writer

Use clang-format

Adhere to openPMD requirements before flushing

For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush.

Maybe open an issue in the openPMD API to allow for a more flexible usage.

Fix an indexing bug

Eliminate dataset preparation for NDScalars

Also fix a bug where particles were named wrongly in checkpoints.

Do not write empty particle datasets

Treat non-existing datasets as empty in reading

Remove prepared datasets

Remove WithWindow struct

Use transform to enable ADIOS1 compression

Remove accidentally versioned files

Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD

Remove traces of the old ADIOS plugin

mostly the word "adios" from comments

Take copies and not references of openPMD handles

Fix autoformatting

Require newer openPMD API

Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH

Add version label to format string only if present

Replace typedefs with using

Remove further indexing bug

in writing particles_info table

Cleanup restart

Remove dataset preparation

Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes.
The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518.

Postpone writing of meta attributes

Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles.
Bug report will follow.

Resolve counting bug during particle writing

Fix whitespaces

Separate ADIOS and HDF5 particle writing strategies

Allow choosing strategy as runtime parameter

Cleanup

Fix openPMD version formatting

Update examples to use openPMD

Refactor passing of filename and extension

Reassemble filename and extension upon opening the series.

Fix some missing includes

Do not skip writing empty datasets

See openPMD PR: openPMD/openPMD-api#529
This allows to write empty datasets

Remove debugging leftovers

Write timeOffset for particles in a standard-compliant way

Do not declare zero-sized arrays

C++ standard requires that array size evaluate to a value greater than zero.

Do not use storeChunk on empty datasets

centralize initialization of thread params from config

Fix undefined identifier in assert statements

Error passes silently in release builds.

Pass JSON config to openPMD::Series ctor

Do not copy Series object

see openPMD/openPMD-api#534

Allow NULL as configuration infix to denote empty string

Adapt to changes in pmacc etc.

Enable use of group-based layout as well

Requires keeping openPMD Series open. Since openPMD currently has no
explicit call to close a file, we implement this only for group-based
layout for now.

Do not use deprecated Series::setSoftwareVersion call

Apply commit b797c0a to openPMD backend

Formatting in .cfg files

Fix an uninitialized value and an indexing bug

Implement reviewers' comments concerning CMakeLists

Fix some bugs and remove unneeded parameters

Some ADIOS-specific parameters have been remove that haven't been
implemented anyway and are going to be implemented via JSON.

Don't set compression if set to "none".

Also, I missed a small part when porting b797c0a to this plugin.

Add documentation for openPMD plugin [WIP]

Further write documentation

Rename data preparation strategies

adios -> doubleBuffer
hdf5  -> mappedMemory
Old names may still be used alternatively.

Fix indexing bug for non domain-bound fields

Fix indexing order in fields

Implement Reviewer's comments

Remove workaround for ADIOS1 backend

fix attributes

Mesh record component positions and unitSI
Iteration: time
ParticleAttribute unitDimension

Implement reviewer's comments (2)

Update copyright headers of files changed

Add compression to openPMD TBG example

Use openPMD::getVersion

remove unnecessary parentheses
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants