Skip to content

Conversation

@franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented Mar 12, 2025

This tries removing the major of two WIP restrictions for random-access reading of variable-encoded Series by preparsing.

TODO

  • Docs
  • Testing
  • Forward engine parameters to the preparsing engine as well

Discussion: The ADIOS2 backend now contains two implementations for preparsing:

  • The old implementation: Only /data/snapshot is preparsed, metadata is otherwise considered to be fully static. Advantage: The file needs to be opened in adios2::Mode::Read for preparsing only on a single rank, only the collected values for /data/snapshot are distributed in a single MPI operation at the end.
  • The new implementation: Full support for reading modifiable attributes. Shows the correct values for attributes such as /data/time and has (restricted) support for groups that are present only in some steps (as long as they only consist of attributes and not arrays, i.e. as long as they are constant components). Disadvantage: Requires collectively reading the entire dataset and keeping the preparsed attributes for all steps in memory.

Hence:

  • Should we keep the old implementation around for its performance benefits when the capabilities of the new implementation are not needed? – No, the old implementation is removed.

@franzpoeschel franzpoeschel changed the title Attribute preparsing for variable-encoded files in ReadRandomAccess mode: Support modifiable attributes WIP Attribute preparsing for variable-encoded files in ReadRandomAccess mode: Support modifiable attributes Mar 12, 2025
@franzpoeschel franzpoeschel force-pushed the topic-preparse-all-attributes branch from c5c9726 to 5eaf97b Compare March 12, 2025 16:45
@franzpoeschel franzpoeschel changed the title WIP Attribute preparsing for variable-encoded files in ReadRandomAccess mode: Support modifiable attributes Attribute preparsing for variable-encoded files in ReadRandomAccess mode: Support modifiable attributes Mar 14, 2025
@franzpoeschel franzpoeschel requested a review from ax3l March 14, 2025 13:26
@franzpoeschel franzpoeschel enabled auto-merge (squash) March 24, 2025 13:09
retrigger CI
@franzpoeschel franzpoeschel force-pushed the topic-preparse-all-attributes branch from d98599b to 361363a Compare March 25, 2025 13:38
@franzpoeschel franzpoeschel merged commit c484088 into openPMD:dev Mar 25, 2025
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant