Skip to content

ADIOS2: more fine-grained control for file endings #1218

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Aug 9, 2022
Merged

ADIOS2: more fine-grained control for file endings #1218

merged 7 commits into from
Aug 9, 2022

Conversation

@franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented Mar 4, 2022 •
edited by ax3l
Loading

Close #1205

TODO:

  • Frontend implementation
  • Backend implementation
  • Ensure that ideally the filename on disk is the same as specified by the user in most cases
  • Testing
  • Merge Clearly fail when users select a wrong backend #1214 first (they would conflict else)
  • What to do for Series("asdf.bp5", Access::CREATE, "adios2.engine.type = \"file\"")?
    Current implementation: Accept this in the backend, but warn the user that the file engine will decide the BP version independent of the extension: [ADIOS2] Specified explicit ending '.bp5' in combination with generic file engine 'file'. ADIOS2 will pick a default file ending independent of specified suffix. (E.g. 'simData.bp5' might actually be written as a BP4 dataset.)
  • Test that Series("asdf.bp5", Access::CREATE, "backend = \"adios2\"") is handled correctly
  • Merge Fixes for BP5 engine #1215 first and delete those commits from this branch
  • Documentation
  • Merge BP5 tests: Only run them if ADIOS2 has BP5 #1262 first
  • extend openPMD::getFileExtensions()
  • Merge Distinguish all char types in ADIOS2 backend  #1275 first

@franzpoeschel franzpoeschel force-pushed the adios2-file-endings branch 3 times, most recently from 934b5fc to 5eb6d2d Compare March 7, 2022 13:50
@franzpoeschel
Copy link
Contributor Author

franzpoeschel commented Mar 7, 2022

Current approach: Each ADIOS2 engine has a set of accepted file endings and a default file ending. If the user specifies any of the accepted endings, it will be used. If no ending is specified or any other ending is specified, the default ending will be used.
E.g.: Engine "file": default ending "bp", accepted endings "bp", "bp3", "bp4", "bp5"
Engine "bp4": default ending "bp4", accepted endings "bp", "bp3", "bp4", "bp5"

@franzpoeschel franzpoeschel force-pushed the adios2-file-endings branch 3 times, most recently from a85c7cf to 067a9a9 Compare March 10, 2022 14:39
@franzpoeschel
Copy link
Contributor Author

franzpoeschel commented Mar 11, 2022 •
edited
Loading

This PR now has some BP5 fixes that I'll need to factor out.

@franzpoeschel franzpoeschel force-pushed the adios2-file-endings branch 5 times, most recently from 665e4ff to 6aa0ba0 Compare March 15, 2022 16:43
@franzpoeschel franzpoeschel changed the title [WIP] ADIOS2: more fine-grained control for file endings ADIOS2: more fine-grained control for file endings Mar 16, 2022
@franzpoeschel franzpoeschel requested a review from ax3l March 16, 2022 16:07
@franzpoeschel franzpoeschel removed the request for review from ax3l March 16, 2022 16:08
@ax3l
Copy link
Member

ax3l commented Apr 15, 2022

@franzpoeschel can you please rebase this? Looks like all predicate PRs are merged now :)

@ax3l ax3l self-requested a review April 15, 2022 17:21
@ax3l ax3l self-assigned this Apr 15, 2022
@franzpoeschel franzpoeschel force-pushed the adios2-file-endings branch 2 times, most recently from 9a8dc96 to ae94962 Compare April 29, 2022 13:09
@franzpoeschel franzpoeschel mentioned this pull request May 3, 2022
Merged
@franzpoeschel franzpoeschel force-pushed the adios2-file-endings branch 3 times, most recently from a6cc8c1 to 36995ea Compare May 16, 2022 12:25
@franzpoeschel franzpoeschel mentioned this pull request May 16, 2022
Merged
@franzpoeschel franzpoeschel force-pushed the adios2-file-endings branch 2 times, most recently from 62592dd to 855562a Compare May 16, 2022 13:33
@franzpoeschel franzpoeschel force-pushed the adios2-file-endings branch 2 times, most recently from 06fda47 to 261f16b Compare May 17, 2022 08:54
This was referenced May 17, 2022
Merged
Open
@franzpoeschel franzpoeschel force-pushed the adios2-file-endings branch from e767aef to 827d991 Compare May 31, 2022 11:36
@franzpoeschel franzpoeschel force-pushed the adios2-file-endings branch from 2cd8de4 to 4143658 Compare June 8, 2022 14:09
@franzpoeschel franzpoeschel force-pushed the adios2-file-endings branch from 6092e24 to 3fd9b75 Compare July 5, 2022 09:44
This was referenced Jul 12, 2022
Merged
Merged
@ax3l
Copy link
Member

ax3l commented Aug 9, 2022

CI looks good, but I'll run it again to make sure it's up-to-date.

@ax3l ax3l closed this Aug 9, 2022
@ax3l ax3l reopened this Aug 9, 2022
ax3l
ax3l approved these changes Aug 9, 2022
View reviewed changes
Copy link
Member

@ax3l ax3l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks a lot! 👍

@ax3l ax3l enabled auto-merge (squash) August 9, 2022 21:19
@ax3l ax3l added this to the 0.15.0 milestone Aug 9, 2022
@ax3l ax3l merged commit 4cf4064 into openPMD:dev Aug 9, 2022
eschnett added a commit to eschnett/openPMD-api that referenced this pull request Nov 11, 2022
@eschnett
Merge branch 'dev' into eschnett/julia-bindings
e6a18e7 
* dev: (70 commits)
  Docs: Recommend Static Build for Superbuilds (openPMD#1325)
  Python 3.11 (openPMD#1323)
  pybind11: v2.10.1+ (openPMD#1322)
  Add Attribute::getOptional<T>() and use to add some more dynamic datatype conversions at read time (openPMD#1278)
  Mapping between ADIOS steps and openPMD iterations (openPMD#949)
  Deprecate shareRaw (openPMD#1229)
  Fix append mode double attributes (openPMD#1302)
  Constant scalars: Don't flush double (openPMD#1315)
  Remove caching cmake vars (openPMD#1313)
  [pre-commit.ci] pre-commit autoupdate (openPMD#1311)
  storeChunk: Add an overload for shared_ptr<T[]> (openPMD#1296)
  Fix `operationAsString` Export (openPMD#1309)
  ADIOS2: more fine-grained control for file endings (openPMD#1218)
  [pre-commit.ci] pre-commit autoupdate (openPMD#1307)
  Fix file existence check in parallel tests (openPMD#1303)
  ADIOS2: Flush to disk within a step (openPMD#1207)
  [pre-commit.ci] pre-commit autoupdate (openPMD#1304)
  [pre-commit.ci] pre-commit autoupdate (openPMD#1295)
  Update catch2 to v2.13.9 (openPMD#1299)
  [pre-commit.ci] pre-commit autoupdate (openPMD#1292)
  ...

# Conflicts:
#	.github/workflows/linux.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ax3l ax3l ax3l approved these changes

Assignees

@ax3l ax3l

Labels

backend: ADIOS2

Projects

None yet

Milestone

0.15.0

Development

Successfully merging this pull request may close these issues.

Distinguish BP versions in file endings

2 participants

@franzpoeschel @ax3l