-
Notifications
You must be signed in to change notification settings - Fork 54
Allow flushing before defining position and positionOffset components of particle species #518
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes. The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518.
This will ensure openPMD specific requirements for those records at flush time as soon as those records are present -- rather than at creation time of the particle species object. Currently missing is any kind of check that those records are present before closing the series.
644b7ef to
4bdd73f
Compare
|
Thank you, great this gets addressed. I think in general that we should remove the creation logic from the objects and divide "openPMD objects" from "required and default attribute creation" in our implementation. This is too interwinded still and causes issues. Furthermore, I think we should relax what happens on
Good point. Maybe we just need an explicit |
|
Before we merge this, did you verify the created files still pass the validator? https://github.com/openPMD/openPMD-validator |
|
Did that just now, the files yield exactly the same errors as those on the dev branch do, so I guess there's further things to investigate. I checked the JSON files with diff now and there are actually differences for the |
|
Ok, let's do that in a follow-up PR. note: |
Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes. The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518.
Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes. The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518.
Create openPMDWriter based on ADIOSWriter (WIP) Adapt CountParticles partially to openPMD Adapt WriteSpecies to openPMD Translate WriteMeta.hpp to openPMD Add function "asStandardVector()" Translate NDScalars.hpp to openPMD Adapt everything to openPMD except for the main file (openPMDWriter.hpp) Adapt openPMDWriter.hpp to openPMD WIP Change management of openPMD Series Further adapt openPMDWriter to openPMD Add openPMD to CMake Add openPMD to version formatting Properly acquaint openPMD plugin with build and help system Make openPMD namespacing explicit Remove adios-specific fields from ThreadParams First successfull run with LaserWakeField example Cleanup Use clang-format Update licensing information Separate basename and filename extension into two separate help parameters Refactor dataset creation Use Mesh::setTimeOffset() template Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch. Clean up some leftovers Use ASCII characters to spell my name Remove unnecessary whitespaces Adapt to removal of pmacc::forward Remove accidentally versioned config file Make checkpoints usable Fix a number of bugs concerning the reading of checkpoints Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation. Remove CollectFieldsSizes Legacy from ADIOS writer, not necessary in openPMD. Remove ParticleAttributeSize and openPMDCountParticles Legacy from ADIOS Writer Use clang-format Adhere to openPMD requirements before flushing For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush. Maybe open an issue in the openPMD API to allow for a more flexible usage. Fix an indexing bug Eliminate dataset preparation for NDScalars Also fix a bug where particles were named wrongly in checkpoints. Do not write empty particle datasets Treat non-existing datasets as empty in reading Remove prepared datasets Remove WithWindow struct Use transform to enable ADIOS1 compression Remove accidentally versioned files Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD Remove traces of the old ADIOS plugin mostly the word "adios" from comments Take copies and not references of openPMD handles Fix autoformatting Require newer openPMD API Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH Add version label to format string only if present Replace typedefs with using Remove further indexing bug in writing particles_info table Cleanup restart Remove dataset preparation Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes. The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518. Postpone writing of meta attributes Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles. Bug report will follow. Resolve counting bug during particle writing Fix whitespaces Separate ADIOS and HDF5 particle writing strategies Allow choosing strategy as runtime parameter Cleanup Fix openPMD version formatting Update examples to use openPMD Refactor passing of filename and extension Reassemble filename and extension upon opening the series. Fix some missing includes Do not skip writing empty datasets See openPMD PR: openPMD/openPMD-api#529 This allows to write empty datasets Remove debugging leftovers Write timeOffset for particles in a standard-compliant way Do not declare zero-sized arrays C++ standard requires that array size evaluate to a value greater than zero. Do not use storeChunk on empty datasets centralize initialization of thread params from config Fix undefined identifier in assert statements Error passes silently in release builds. Pass JSON config to openPMD::Series ctor Do not copy Series object see openPMD/openPMD-api#534 Allow NULL as configuration infix to denote empty string Adapt to changes in pmacc etc. Enable use of group-based layout as well Requires keeping openPMD Series open. Since openPMD currently has no explicit call to close a file, we implement this only for group-based layout for now. Do not use deprecated Series::setSoftwareVersion call Apply commit b797c0a to openPMD backend Formatting in .cfg files Fix an uninitialized value and an indexing bug Implement reviewers' comments concerning CMakeLists
Create openPMDWriter based on ADIOSWriter (WIP) Adapt CountParticles partially to openPMD Adapt WriteSpecies to openPMD Translate WriteMeta.hpp to openPMD Add function "asStandardVector()" Translate NDScalars.hpp to openPMD Adapt everything to openPMD except for the main file (openPMDWriter.hpp) Adapt openPMDWriter.hpp to openPMD WIP Change management of openPMD Series Further adapt openPMDWriter to openPMD Add openPMD to CMake Add openPMD to version formatting Properly acquaint openPMD plugin with build and help system Make openPMD namespacing explicit Remove adios-specific fields from ThreadParams First successfull run with LaserWakeField example Cleanup Use clang-format Update licensing information Separate basename and filename extension into two separate help parameters Refactor dataset creation Use Mesh::setTimeOffset() template Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch. Clean up some leftovers Use ASCII characters to spell my name Remove unnecessary whitespaces Adapt to removal of pmacc::forward Remove accidentally versioned config file Make checkpoints usable Fix a number of bugs concerning the reading of checkpoints Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation. Remove CollectFieldsSizes Legacy from ADIOS writer, not necessary in openPMD. Remove ParticleAttributeSize and openPMDCountParticles Legacy from ADIOS Writer Use clang-format Adhere to openPMD requirements before flushing For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush. Maybe open an issue in the openPMD API to allow for a more flexible usage. Fix an indexing bug Eliminate dataset preparation for NDScalars Also fix a bug where particles were named wrongly in checkpoints. Do not write empty particle datasets Treat non-existing datasets as empty in reading Remove prepared datasets Remove WithWindow struct Use transform to enable ADIOS1 compression Remove accidentally versioned files Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD Remove traces of the old ADIOS plugin mostly the word "adios" from comments Take copies and not references of openPMD handles Fix autoformatting Require newer openPMD API Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH Add version label to format string only if present Replace typedefs with using Remove further indexing bug in writing particles_info table Cleanup restart Remove dataset preparation Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes. The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518. Postpone writing of meta attributes Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles. Bug report will follow. Resolve counting bug during particle writing Fix whitespaces Separate ADIOS and HDF5 particle writing strategies Allow choosing strategy as runtime parameter Cleanup Fix openPMD version formatting Update examples to use openPMD Refactor passing of filename and extension Reassemble filename and extension upon opening the series. Fix some missing includes Do not skip writing empty datasets See openPMD PR: openPMD/openPMD-api#529 This allows to write empty datasets Remove debugging leftovers Write timeOffset for particles in a standard-compliant way Do not declare zero-sized arrays C++ standard requires that array size evaluate to a value greater than zero. Do not use storeChunk on empty datasets centralize initialization of thread params from config Fix undefined identifier in assert statements Error passes silently in release builds. Pass JSON config to openPMD::Series ctor Do not copy Series object see openPMD/openPMD-api#534 Allow NULL as configuration infix to denote empty string Adapt to changes in pmacc etc. Enable use of group-based layout as well Requires keeping openPMD Series open. Since openPMD currently has no explicit call to close a file, we implement this only for group-based layout for now. Do not use deprecated Series::setSoftwareVersion call Apply commit b797c0a to openPMD backend Formatting in .cfg files Fix an uninitialized value and an indexing bug Implement reviewers' comments concerning CMakeLists
Create openPMDWriter based on ADIOSWriter (WIP) Adapt CountParticles partially to openPMD Adapt WriteSpecies to openPMD Translate WriteMeta.hpp to openPMD Add function "asStandardVector()" Translate NDScalars.hpp to openPMD Adapt everything to openPMD except for the main file (openPMDWriter.hpp) Adapt openPMDWriter.hpp to openPMD WIP Change management of openPMD Series Further adapt openPMDWriter to openPMD Add openPMD to CMake Add openPMD to version formatting Properly acquaint openPMD plugin with build and help system Make openPMD namespacing explicit Remove adios-specific fields from ThreadParams First successfull run with LaserWakeField example Cleanup Use clang-format Update licensing information Separate basename and filename extension into two separate help parameters Refactor dataset creation Use Mesh::setTimeOffset() template Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch. Clean up some leftovers Use ASCII characters to spell my name Remove unnecessary whitespaces Adapt to removal of pmacc::forward Remove accidentally versioned config file Make checkpoints usable Fix a number of bugs concerning the reading of checkpoints Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation. Remove CollectFieldsSizes Legacy from ADIOS writer, not necessary in openPMD. Remove ParticleAttributeSize and openPMDCountParticles Legacy from ADIOS Writer Use clang-format Adhere to openPMD requirements before flushing For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush. Maybe open an issue in the openPMD API to allow for a more flexible usage. Fix an indexing bug Eliminate dataset preparation for NDScalars Also fix a bug where particles were named wrongly in checkpoints. Do not write empty particle datasets Treat non-existing datasets as empty in reading Remove prepared datasets Remove WithWindow struct Use transform to enable ADIOS1 compression Remove accidentally versioned files Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD Remove traces of the old ADIOS plugin mostly the word "adios" from comments Take copies and not references of openPMD handles Fix autoformatting Require newer openPMD API Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH Add version label to format string only if present Replace typedefs with using Remove further indexing bug in writing particles_info table Cleanup restart Remove dataset preparation Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes. The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518. Postpone writing of meta attributes Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles. Bug report will follow. Resolve counting bug during particle writing Fix whitespaces Separate ADIOS and HDF5 particle writing strategies Allow choosing strategy as runtime parameter Cleanup Fix openPMD version formatting Update examples to use openPMD Refactor passing of filename and extension Reassemble filename and extension upon opening the series. Fix some missing includes Do not skip writing empty datasets See openPMD PR: openPMD/openPMD-api#529 This allows to write empty datasets Remove debugging leftovers Write timeOffset for particles in a standard-compliant way Do not declare zero-sized arrays C++ standard requires that array size evaluate to a value greater than zero. Do not use storeChunk on empty datasets centralize initialization of thread params from config Fix undefined identifier in assert statements Error passes silently in release builds. Pass JSON config to openPMD::Series ctor Do not copy Series object see openPMD/openPMD-api#534 Allow NULL as configuration infix to denote empty string Adapt to changes in pmacc etc. Enable use of group-based layout as well Requires keeping openPMD Series open. Since openPMD currently has no explicit call to close a file, we implement this only for group-based layout for now. Do not use deprecated Series::setSoftwareVersion call Apply commit b797c0a to openPMD backend Formatting in .cfg files Fix an uninitialized value and an indexing bug Implement reviewers' comments concerning CMakeLists
Create openPMDWriter based on ADIOSWriter (WIP) Adapt CountParticles partially to openPMD Adapt WriteSpecies to openPMD Translate WriteMeta.hpp to openPMD Add function "asStandardVector()" Translate NDScalars.hpp to openPMD Adapt everything to openPMD except for the main file (openPMDWriter.hpp) Adapt openPMDWriter.hpp to openPMD WIP Change management of openPMD Series Further adapt openPMDWriter to openPMD Add openPMD to CMake Add openPMD to version formatting Properly acquaint openPMD plugin with build and help system Make openPMD namespacing explicit Remove adios-specific fields from ThreadParams First successfull run with LaserWakeField example Cleanup Use clang-format Update licensing information Separate basename and filename extension into two separate help parameters Refactor dataset creation Use Mesh::setTimeOffset() template Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch. Clean up some leftovers Use ASCII characters to spell my name Remove unnecessary whitespaces Adapt to removal of pmacc::forward Remove accidentally versioned config file Make checkpoints usable Fix a number of bugs concerning the reading of checkpoints Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation. Remove CollectFieldsSizes Legacy from ADIOS writer, not necessary in openPMD. Remove ParticleAttributeSize and openPMDCountParticles Legacy from ADIOS Writer Use clang-format Adhere to openPMD requirements before flushing For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush. Maybe open an issue in the openPMD API to allow for a more flexible usage. Fix an indexing bug Eliminate dataset preparation for NDScalars Also fix a bug where particles were named wrongly in checkpoints. Do not write empty particle datasets Treat non-existing datasets as empty in reading Remove prepared datasets Remove WithWindow struct Use transform to enable ADIOS1 compression Remove accidentally versioned files Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD Remove traces of the old ADIOS plugin mostly the word "adios" from comments Take copies and not references of openPMD handles Fix autoformatting Require newer openPMD API Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH Add version label to format string only if present Replace typedefs with using Remove further indexing bug in writing particles_info table Cleanup restart Remove dataset preparation Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes. The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518. Postpone writing of meta attributes Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles. Bug report will follow. Resolve counting bug during particle writing Fix whitespaces Separate ADIOS and HDF5 particle writing strategies Allow choosing strategy as runtime parameter Cleanup Fix openPMD version formatting Update examples to use openPMD Refactor passing of filename and extension Reassemble filename and extension upon opening the series. Fix some missing includes Do not skip writing empty datasets See openPMD PR: openPMD/openPMD-api#529 This allows to write empty datasets Remove debugging leftovers Write timeOffset for particles in a standard-compliant way Do not declare zero-sized arrays C++ standard requires that array size evaluate to a value greater than zero. Do not use storeChunk on empty datasets centralize initialization of thread params from config Fix undefined identifier in assert statements Error passes silently in release builds. Pass JSON config to openPMD::Series ctor Do not copy Series object see openPMD/openPMD-api#534 Allow NULL as configuration infix to denote empty string Adapt to changes in pmacc etc. Enable use of group-based layout as well Requires keeping openPMD Series open. Since openPMD currently has no explicit call to close a file, we implement this only for group-based layout for now. Do not use deprecated Series::setSoftwareVersion call Apply commit b797c0a to openPMD backend Formatting in .cfg files Fix an uninitialized value and an indexing bug Implement reviewers' comments concerning CMakeLists
Create openPMDWriter based on ADIOSWriter (WIP) Adapt CountParticles partially to openPMD Adapt WriteSpecies to openPMD Translate WriteMeta.hpp to openPMD Add function "asStandardVector()" Translate NDScalars.hpp to openPMD Adapt everything to openPMD except for the main file (openPMDWriter.hpp) Adapt openPMDWriter.hpp to openPMD WIP Change management of openPMD Series Further adapt openPMDWriter to openPMD Add openPMD to CMake Add openPMD to version formatting Properly acquaint openPMD plugin with build and help system Make openPMD namespacing explicit Remove adios-specific fields from ThreadParams First successfull run with LaserWakeField example Cleanup Use clang-format Update licensing information Separate basename and filename extension into two separate help parameters Refactor dataset creation Use Mesh::setTimeOffset() template Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch. Clean up some leftovers Use ASCII characters to spell my name Remove unnecessary whitespaces Adapt to removal of pmacc::forward Remove accidentally versioned config file Make checkpoints usable Fix a number of bugs concerning the reading of checkpoints Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation. Remove CollectFieldsSizes Legacy from ADIOS writer, not necessary in openPMD. Remove ParticleAttributeSize and openPMDCountParticles Legacy from ADIOS Writer Use clang-format Adhere to openPMD requirements before flushing For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush. Maybe open an issue in the openPMD API to allow for a more flexible usage. Fix an indexing bug Eliminate dataset preparation for NDScalars Also fix a bug where particles were named wrongly in checkpoints. Do not write empty particle datasets Treat non-existing datasets as empty in reading Remove prepared datasets Remove WithWindow struct Use transform to enable ADIOS1 compression Remove accidentally versioned files Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD Remove traces of the old ADIOS plugin mostly the word "adios" from comments Take copies and not references of openPMD handles Fix autoformatting Require newer openPMD API Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH Add version label to format string only if present Replace typedefs with using Remove further indexing bug in writing particles_info table Cleanup restart Remove dataset preparation Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes. The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518. Postpone writing of meta attributes Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles. Bug report will follow. Resolve counting bug during particle writing Fix whitespaces Separate ADIOS and HDF5 particle writing strategies Allow choosing strategy as runtime parameter Cleanup Fix openPMD version formatting Update examples to use openPMD Refactor passing of filename and extension Reassemble filename and extension upon opening the series. Fix some missing includes Do not skip writing empty datasets See openPMD PR: openPMD/openPMD-api#529 This allows to write empty datasets Remove debugging leftovers Write timeOffset for particles in a standard-compliant way Do not declare zero-sized arrays C++ standard requires that array size evaluate to a value greater than zero. Do not use storeChunk on empty datasets centralize initialization of thread params from config Fix undefined identifier in assert statements Error passes silently in release builds. Pass JSON config to openPMD::Series ctor Do not copy Series object see openPMD/openPMD-api#534 Allow NULL as configuration infix to denote empty string Adapt to changes in pmacc etc. Enable use of group-based layout as well Requires keeping openPMD Series open. Since openPMD currently has no explicit call to close a file, we implement this only for group-based layout for now. Do not use deprecated Series::setSoftwareVersion call Apply commit b797c0a to openPMD backend Formatting in .cfg files Fix an uninitialized value and an indexing bug Implement reviewers' comments concerning CMakeLists
Create openPMDWriter based on ADIOSWriter (WIP) Adapt CountParticles partially to openPMD Adapt WriteSpecies to openPMD Translate WriteMeta.hpp to openPMD Add function "asStandardVector()" Translate NDScalars.hpp to openPMD Adapt everything to openPMD except for the main file (openPMDWriter.hpp) Adapt openPMDWriter.hpp to openPMD WIP Change management of openPMD Series Further adapt openPMDWriter to openPMD Add openPMD to CMake Add openPMD to version formatting Properly acquaint openPMD plugin with build and help system Make openPMD namespacing explicit Remove adios-specific fields from ThreadParams First successfull run with LaserWakeField example Cleanup Use clang-format Update licensing information Separate basename and filename extension into two separate help parameters Refactor dataset creation Use Mesh::setTimeOffset() template Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch. Clean up some leftovers Use ASCII characters to spell my name Remove unnecessary whitespaces Adapt to removal of pmacc::forward Remove accidentally versioned config file Make checkpoints usable Fix a number of bugs concerning the reading of checkpoints Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation. Remove CollectFieldsSizes Legacy from ADIOS writer, not necessary in openPMD. Remove ParticleAttributeSize and openPMDCountParticles Legacy from ADIOS Writer Use clang-format Adhere to openPMD requirements before flushing For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush. Maybe open an issue in the openPMD API to allow for a more flexible usage. Fix an indexing bug Eliminate dataset preparation for NDScalars Also fix a bug where particles were named wrongly in checkpoints. Do not write empty particle datasets Treat non-existing datasets as empty in reading Remove prepared datasets Remove WithWindow struct Use transform to enable ADIOS1 compression Remove accidentally versioned files Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD Remove traces of the old ADIOS plugin mostly the word "adios" from comments Take copies and not references of openPMD handles Fix autoformatting Require newer openPMD API Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH Add version label to format string only if present Replace typedefs with using Remove further indexing bug in writing particles_info table Cleanup restart Remove dataset preparation Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes. The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518. Postpone writing of meta attributes Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles. Bug report will follow. Resolve counting bug during particle writing Fix whitespaces Separate ADIOS and HDF5 particle writing strategies Allow choosing strategy as runtime parameter Cleanup Fix openPMD version formatting Update examples to use openPMD Refactor passing of filename and extension Reassemble filename and extension upon opening the series. Fix some missing includes Do not skip writing empty datasets See openPMD PR: openPMD/openPMD-api#529 This allows to write empty datasets Remove debugging leftovers Write timeOffset for particles in a standard-compliant way Do not declare zero-sized arrays C++ standard requires that array size evaluate to a value greater than zero. Do not use storeChunk on empty datasets centralize initialization of thread params from config Fix undefined identifier in assert statements Error passes silently in release builds. Pass JSON config to openPMD::Series ctor Do not copy Series object see openPMD/openPMD-api#534 Allow NULL as configuration infix to denote empty string Adapt to changes in pmacc etc. Enable use of group-based layout as well Requires keeping openPMD Series open. Since openPMD currently has no explicit call to close a file, we implement this only for group-based layout for now. Do not use deprecated Series::setSoftwareVersion call Apply commit b797c0a to openPMD backend Formatting in .cfg files Fix an uninitialized value and an indexing bug Implement reviewers' comments concerning CMakeLists Fix some bugs and remove unneeded parameters Some ADIOS-specific parameters have been remove that haven't been implemented anyway and are going to be implemented via JSON. Don't set compression if set to "none". Also, I missed a small part when porting b797c0a to this plugin. Add documentation for openPMD plugin [WIP] Further write documentation Rename data preparation strategies adios -> doubleBuffer hdf5 -> mappedMemory Old names may still be used alternatively. Fix indexing bug for non domain-bound fields Fix indexing order in fields Implement Reviewer's comments Remove workaround for ADIOS1 backend fix attributes Mesh record component positions and unitSI Iteration: time ParticleAttribute unitDimension Implement reviewer's comments (2) Update copyright headers of files changed Add compression to openPMD TBG example Use openPMD::getVersion remove unnecessary parentheses
Create openPMDWriter based on ADIOSWriter (WIP) Adapt CountParticles partially to openPMD Adapt WriteSpecies to openPMD Translate WriteMeta.hpp to openPMD Add function "asStandardVector()" Translate NDScalars.hpp to openPMD Adapt everything to openPMD except for the main file (openPMDWriter.hpp) Adapt openPMDWriter.hpp to openPMD WIP Change management of openPMD Series Further adapt openPMDWriter to openPMD Add openPMD to CMake Add openPMD to version formatting Properly acquaint openPMD plugin with build and help system Make openPMD namespacing explicit Remove adios-specific fields from ThreadParams First successfull run with LaserWakeField example Cleanup Use clang-format Update licensing information Separate basename and filename extension into two separate help parameters Refactor dataset creation Use Mesh::setTimeOffset() template Causes a linker error with the current version of openPMD API, a fix will be merged soon on the dev branch. Clean up some leftovers Use ASCII characters to spell my name Remove unnecessary whitespaces Adapt to removal of pmacc::forward Remove accidentally versioned config file Make checkpoints usable Fix a number of bugs concerning the reading of checkpoints Still problematic is the attribute "timeOffset", currently mitigated by uncommenting the sanity check in the openPMD API. Needs further investigation. Remove CollectFieldsSizes Legacy from ADIOS writer, not necessary in openPMD. Remove ParticleAttributeSize and openPMDCountParticles Legacy from ADIOS Writer Use clang-format Adhere to openPMD requirements before flushing For a given particle species, the openPMD API requires that required records (such as "position", "positionOffset") and their contained components be defined (not necessarily written). Make sure to define all such datasets before issuing the first flush. Maybe open an issue in the openPMD API to allow for a more flexible usage. Fix an indexing bug Eliminate dataset preparation for NDScalars Also fix a bug where particles were named wrongly in checkpoints. Do not write empty particle datasets Treat non-existing datasets as empty in reading Remove prepared datasets Remove WithWindow struct Use transform to enable ADIOS1 compression Remove accidentally versioned files Rename LoadParticleAttributesFromADIOS to LoadParticleAttributesFromOpenPMD Remove traces of the old ADIOS plugin mostly the word "adios" from comments Take copies and not references of openPMD handles Fix autoformatting Require newer openPMD API Also add ADIOS2_ROOT to CMAKE_PREFIX_PATH Add version label to format string only if present Replace typedefs with using Remove further indexing bug in writing particles_info table Cleanup restart Remove dataset preparation Commit 0b50561 reintroduced a reduced form of dataset preparation in order to adhere to requirements (restrictions) of the openPMD API. This workaround results in dummy dataset writes (likely a bug in the openPMD API), hence this commit reverts those changes. The corresponding pull request in the openPMD API to relax this restriction can be found at openPMD/openPMD-api#518. Postpone writing of meta attributes Due to a bug in the ADIOS1 backend of the openPMD API, the first dataset needs to be written before the first flush. This works around the problem by first defining all the fields and particles. Bug report will follow. Resolve counting bug during particle writing Fix whitespaces Separate ADIOS and HDF5 particle writing strategies Allow choosing strategy as runtime parameter Cleanup Fix openPMD version formatting Update examples to use openPMD Refactor passing of filename and extension Reassemble filename and extension upon opening the series. Fix some missing includes Do not skip writing empty datasets See openPMD PR: openPMD/openPMD-api#529 This allows to write empty datasets Remove debugging leftovers Write timeOffset for particles in a standard-compliant way Do not declare zero-sized arrays C++ standard requires that array size evaluate to a value greater than zero. Do not use storeChunk on empty datasets centralize initialization of thread params from config Fix undefined identifier in assert statements Error passes silently in release builds. Pass JSON config to openPMD::Series ctor Do not copy Series object see openPMD/openPMD-api#534 Allow NULL as configuration infix to denote empty string Adapt to changes in pmacc etc. Enable use of group-based layout as well Requires keeping openPMD Series open. Since openPMD currently has no explicit call to close a file, we implement this only for group-based layout for now. Do not use deprecated Series::setSoftwareVersion call Apply commit b797c0a to openPMD backend Formatting in .cfg files Fix an uninitialized value and an indexing bug Implement reviewers' comments concerning CMakeLists Fix some bugs and remove unneeded parameters Some ADIOS-specific parameters have been remove that haven't been implemented anyway and are going to be implemented via JSON. Don't set compression if set to "none". Also, I missed a small part when porting b797c0a to this plugin. Add documentation for openPMD plugin [WIP] Further write documentation Rename data preparation strategies adios -> doubleBuffer hdf5 -> mappedMemory Old names may still be used alternatively. Fix indexing bug for non domain-bound fields Fix indexing order in fields Implement Reviewer's comments Remove workaround for ADIOS1 backend fix attributes Mesh record component positions and unitSI Iteration: time ParticleAttribute unitDimension Implement reviewer's comments (2) Update copyright headers of files changed Add compression to openPMD TBG example Use openPMD::getVersion remove unnecessary parentheses
Issue The fact that the class
ParticleSpecieswill enforce openPMD requirements currently leads to the awkward situation that those records have to be defined usingRecordComponent::resetDataset( … )before issuing the first flush on theSeriesobject. A laterRecordComponent::storeChunk( … )is fine, but the datasets need to be defined up front.This is (1) an awkward pitfall, (2) leads to rather confusing errors if done wrongly (
what(): A Record can not be written without any contained RecordComponents: position) and (3) leads to ugly workarounds in workflows where the single records of a particle species are defined and written one after another (e.g. the openPMD plugin for picongpu that I am currently working on).Suggested Change Move these two lines into the
flushmethod override ofParticleSpecies, but do not create the records if not present.Open questions The above solution will ensure openPMD requirements for position[Offset] records if the user remembers to create them. I have not (yet) implemented an alternative check if the user forgets to create those records. The logical location for such a check would be the destructor, but throwing exceptions in destructors is discouraged.
First commit adds new tests (and removes some existing ones), second commit will implement the suggested change.