-
Notifications
You must be signed in to change notification settings - Fork 5
Raw Echoview export ingestion and batch processing #257
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
brandynlucca
merged 49 commits into
OSOceanAcoustics:main
from
brandynlucca:WIP_nasc_directory_ingestion
Aug 28, 2024
Merged
Raw Echoview export ingestion and batch processing #257
brandynlucca
merged 49 commits into
OSOceanAcoustics:main
from
brandynlucca:WIP_nasc_directory_ingestion
Aug 28, 2024
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…IP_nasc_directory_ingestion
for more information, see https://pre-commit.ci
…b.com/brandynlucca/echopop into WIP_result_summary_output_enhancement
for more information, see https://pre-commit.ci
…ndynlucca/echopop into WIP_nasc_directory_ingestion
for more information, see https://pre-commit.ci
…ndynlucca/echopop into WIP_nasc_directory_ingestion
updates: - [github.com/psf/black: 24.4.2 → 24.8.0](psf/black@24.4.2...24.8.0) - [github.com/PyCQA/flake8: 7.1.0 → 7.1.1](PyCQA/flake8@7.1.0...7.1.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
leewujung
reviewed
Aug 9, 2024
leewujung
reviewed
Aug 9, 2024
leewujung
reviewed
Aug 9, 2024
leewujung
reviewed
Aug 9, 2024
for more information, see https://pre-commit.ci
…ndynlucca/echopop into WIP_nasc_directory_ingestion
for more information, see https://pre-commit.ci
…ndynlucca/echopop into WIP_nasc_directory_ingestion
related file creation
for more information, see https://pre-commit.ci
…ndynlucca/echopop into WIP_nasc_directory_ingestion
for more information, see https://pre-commit.ci
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes a variety of new features for ingesting and batch processing Echoview exports to both 1) write a new
xlsxfile with the consolidated vertically integrated (NASC) exports for age-1+ and age-2+ datasets and 2) directly update theSurveyclass object to read the updated data. These changes include:New batch loader sub-module
batch_loadsub-module has been added to theutilssub-package. This is called when theSurvey-class object is initialized, which adds additional arguments to the initialization. Alternatively, the primary batch processing function,bach_read_echoview_exports(...), can be invoked independent of theSurveyobject.Data parameterization changes
initialization_config.yml(nasc_exports) introducing various keys for parameterizing the outputs.survey_year_****_config.ymlenabling filepaths to echogram regions (i.e.export_regions).core.pyrequired for data validation (ECHOVIEW_EXPORT_MAPandREGION_EXPORT_MAP).NASC_all_agesvsNASC_no_age1. The*_all_agesand*_no_age1have now been generalized as appended 'tags' to age group-specific values that are then converted to shared column names (i.e.,nasc,stratum_num, andhaul_num) depending on theexclude_age1argument value. This necessitated some changes to theloadsub-module in theutilssub-package.Housekeeping changes
.pre-commit-config.yamland the creation of an associated.codespell-ignore-words.txtthat is compatible with misspelled column names in data sources.Note
NASC constructionoperation in EchoPro) and the pre-generated consolidated export files we have received previously. This is being diagnosed, but appears to be due to differences in the source export files rather than anything within the code.