Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
b126399
Create test script
brandynlucca Dec 5, 2024
0e512e0
Compability adjustments for 2017
brandynlucca Dec 5, 2024
bb95f1b
Reindexing change required for 2021 dataset
brandynlucca Dec 5, 2024
96ec6c5
Fix column name parsing when `regex=False` in `pandera` `DataFrame` v…
brandynlucca Dec 6, 2024
d4141f6
Fix `*.csv` filtering
brandynlucca Dec 7, 2024
b62b9c8
Limit options for `Lat_S` and `Lon_S`
brandynlucca Dec 7, 2024
3db6070
Fix to empty/whitespace in `*.xlsx` files
brandynlucca Dec 7, 2024
6b373d0
Upload support files for backwards compatibility testing
brandynlucca Dec 7, 2024
8de8bd3
Index by `"stratum_inpfc"` AND `"transect_num"` to
brandynlucca Dec 9, 2024
e05b1fc
Ensure that `transect_summary` index is properly reset and resolve `p…
brandynlucca Dec 9, 2024
2c82a5f
Diagnosing 2013 survey issues
brandynlucca Dec 9, 2024
9a7e525
Merge branch 'main' of https://github.com/brandynlucca/echopop into b…
brandynlucca Dec 10, 2024
1470d8b
Address mismatched stratum indices across hauls
brandynlucca Dec 10, 2024
f373899
Overhaul to Echoview export ingestion step that correctly reads in gr…
brandynlucca Dec 11, 2024
e0f3c06
Variety of commits for fixing years 2015-2023
brandynlucca Dec 12, 2024
2b3fa08
Update configuration files
brandynlucca Dec 13, 2024
7bd02c0
Change to imported stratum-haul mapping for INPFC
brandynlucca Dec 13, 2024
6c0acd8
Stratification fixes
brandynlucca Dec 13, 2024
9b8d3d5
Update to `test_survey` script used for testing each year
brandynlucca Dec 13, 2024
4ec9524
Add `haul ` as a possible column name for translation
brandynlucca Dec 13, 2024
8fed943
Testing file update for tracking success
brandynlucca Dec 13, 2024
32f3afd
Update to 2013 configuration
brandynlucca Dec 13, 2024
189b886
Further update for test file
brandynlucca Dec 13, 2024
3dec490
Changes to address odd 2012 virtual transect issue
brandynlucca Dec 13, 2024
8a33eb8
Adjustment to how INPFC sheet is read in
brandynlucca Dec 14, 2024
c8fa839
Geostrata latitude sorting fixes monotonic issue
brandynlucca Dec 14, 2024
8ba14b7
Compatibility improvements for 2011
brandynlucca Dec 30, 2024
1a8981a
Merge branch 'main' of https://github.com/brandynlucca/echopop into b…
brandynlucca Jan 7, 2025
3fa5243
Initial spot check to column name validator
brandynlucca Jan 23, 2025
11f6300
Changes to validators to enable flexible column naming schemes
brandynlucca Jan 27, 2025
01a2f9d
Fix to f-string
brandynlucca Jan 27, 2025
141e4a8
Fix to backslash in f-string
brandynlucca Jan 27, 2025
048458f
Add triple-quoting to f-string Literal
brandynlucca Jan 27, 2025
21d105e
Changes to address large abundance estimate deviation
brandynlucca Jan 30, 2025
6fbee49
Enable non-grouped transect region file mapping in configuration YAML
brandynlucca Jan 30, 2025
287687f
Adjustments to transect region file that is called for the 2011 surve…
brandynlucca Jan 31, 2025
a9fbaa8
Adjustment to convex hull cropping method
brandynlucca Jan 31, 2025
7b49b65
Fixes to kriged abundance report values
brandynlucca Feb 1, 2025
b23e278
Stored changes to file investigation script
brandynlucca Feb 3, 2025
977373c
Enable .xls file extension
brandynlucca Feb 3, 2025
2695e16
Revert .xls functionality (temporarily)
brandynlucca Feb 3, 2025
7d7461a
Enable 2011 transect interval filter file ingestion
brandynlucca Apr 23, 2025
8a54aad
Update biodata ingestion (#9)
brandynlucca Apr 23, 2025
cf4c2cf
Merge branch 'OSOceanAcoustics:main' into main
brandynlucca Apr 29, 2025
25a3612
Merge branch 'backwards_compatibility'
brandynlucca Apr 29, 2025
48f8e9c
Fix to `CSVFile` validator
brandynlucca Apr 29, 2025
7d9fe85
Revert consolidated biodata sheet
brandynlucca Apr 29, 2025
90d5eb0
Modified test script and config
brandynlucca Apr 29, 2025
9de3fd2
Bug hotfixes to data loading
brandynlucca Apr 29, 2025
f9aa41f
Test for 2011
brandynlucca Apr 29, 2025
da78a47
Merge pull request #12 from brandynlucca:updated_biodata_loading
brandynlucca Apr 29, 2025
5969e33
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 29, 2025
987b46a
Fix to `test_survey`
brandynlucca Apr 29, 2025
252fbdc
Pre-commit fixes
brandynlucca Apr 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .codespell-ignore-words.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ Calss
calss
Tranect
tranect
notin
4 changes: 3 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@ exclude: |
(?x)^(
echopop/test_data/|
docs/|
project_docs
project_docs|
echopop/file_investigation.py|
echopop/test_survey.py
)
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
Expand Down
107 changes: 107 additions & 0 deletions config_files/initialization_config_2011.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# This YAML file is a configuration file for all
# initialization parameters used in Echopop

---
#####################################################################################################################
# Survey data processing#
########################
# Jolly and Hampton (1990) stratified mean calculation
stratified_survey_mean_parameters:
strata_transect_proportion: 0.75 # percent of Jolly-Hampton transects in each stratum
num_replicates: 10000 # number of resample replicates
mesh_transects_per_latitude: 5 # number of virtual transects per degree latitude in kriging mesh

#####################################################################################################################
# Biological data processing#
########################
# Jolly and Hampton (1990) stratified mean calculation
bio_hake_len_bin: [ # length sequence array 2 - 80 cm. This specifies the bin centers!
2, # start of bin centers
80, # end of bin centers
40 # number of bins in total
]

bio_hake_age_bin: [ # age sequence array: year - age 1-22
1, # start of sequence
22, # end of sequence
22 # number of points in the sequence
]

#####################################################################################################################
# Acoustic transect export processing#
########################
nasc_exports:
export_file_directory: /raw_nasc
nasc_export_directory: /Exports # Acoustic export directory
save_file_template: "{REGION}_NASC_{YEAR}_table_{GROUP}.xlsx"
save_file_sheetname: Sheet1
regions: # Main acoustic data region names (list or single string)
all_ages: [Age-1 Hake, # Age-1+
Age-1 Hake Mix,
Hake,
Hake Mix]
no_age1: [Hake, Hake Mix] # Age-2+
max_transect_spacing: 10.0 # Maximum transect spacing (nmi)
file_columns: [transect_num, region_id, distance_s, distance_e, latitude, longitude,
stratum_num, transect_spacing, layer_mean_depth, layer_height, bottom_depth,
NASC, haul_num]
transect_region_mapping:
parts:
REGION_CLASS:
- pattern: ^[hH](?![a-zA-Z]|1a)
label: Hake
- pattern: ^[hH]1[aA][mM]
label: Age-1 Hake Mix
- pattern: ^[hH]1[aA]
label: Age-1 Hake
- pattern: ^[hH][mM]
label: Hake Mix
- pattern: hake_mix
label: Hake Mix
- pattern: hake
label: Hake
HAUL_NUM:
- pattern: '[0-9]+'
label: None
COUNTRY:
- pattern: ^[cC]
label: CAN
- pattern: ^[uU]
label: US
pattern: "{REGION_CLASS}{HAUL_NUM}{COUNTRY}"
inpfc_strata_region:
CAN: [6]
US: [1, 2, 3, 4, 5]
save_file_template: "{COUNTRY}_{YEAR}_transect_region_haul_{GROUP}.xlsx"
save_file_directory: /Stratification
save_file_sheetname: Sheet1


#####################################################################################################################
# Log-linear regression#
########################
# Target strength (TS) - length (L) regression: TS=m*log10(L)+b
TS_length_regression_parameters:
pacific_hake: # corresponding species text code
number_code: 22500 # species number code
TS_L_slope: 20.0 # the 'm' or 'slope' parameter
TS_L_intercept: -68.0 # the 'b' or 'y-intercept'
length_units: cm # units for L used in regression/relationship

#####################################################################################################################
# Geospatial settings#
########################
geospatial:
init: epsg:4326 # EPSG integer code for geodetic parameter dataset

#####################################################################################################################
# Kriging parameters#
#####################
kriging_parameters:
A0: 6.25 # base area of the grid cell
longitude_reference: -124.78338 # longitude reference for kriging mesh adjustment
longitude_offset: -124.78338 # longitudinal offset for kriging mesh adjustment
latitude_offset: 45.0 # latitudinal offset for kriging mesh adjustment

#####################################################################################################################
...
107 changes: 107 additions & 0 deletions config_files/initialization_config_2012.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# This YAML file is a configuration file for all
# initialization parameters used in Echopop

---
#####################################################################################################################
# Survey data processing#
########################
# Jolly and Hampton (1990) stratified mean calculation
stratified_survey_mean_parameters:
strata_transect_proportion: 0.75 # percent of Jolly-Hampton transects in each stratum
num_replicates: 10000 # number of resample replicates
mesh_transects_per_latitude: 5 # number of virtual transects per degree latitude in kriging mesh

#####################################################################################################################
# Biological data processing#
########################
# Jolly and Hampton (1990) stratified mean calculation
bio_hake_len_bin: [ # length sequence array 2 - 80 cm. This specifies the bin centers!
2, # start of bin centers
80, # end of bin centers
40 # number of bins in total
]

bio_hake_age_bin: [ # age sequence array: year - age 1-22
1, # start of sequence
22, # end of sequence
22 # number of points in the sequence
]

#####################################################################################################################
# Acoustic transect export processing#
########################
nasc_exports:
export_file_directory: /raw_nasc
nasc_export_directory: /Exports # Acoustic export directory
save_file_template: "{REGION}_NASC_{YEAR}_table_{GROUP}.xlsx"
save_file_sheetname: Sheet1
regions: # Main acoustic data region names (list or single string)
all_ages: [Age-1 Hake, # Age-1+
Age-1 Hake Mix,
Hake,
Hake Mix]
no_age1: [Hake, Hake Mix] # Age-2+
max_transect_spacing: 10.0 # Maximum transect spacing (nmi)
file_columns: [transect_num, region_id, distance_s, distance_e, latitude, longitude,
stratum_num, transect_spacing, layer_mean_depth, layer_height, bottom_depth,
NASC, haul_num]
transect_region_mapping:
parts:
REGION_CLASS:
- pattern: ^[hH](?![a-zA-Z]|1a)
label: Hake
- pattern: ^[hH]1[aA][mM]
label: Age-1 Hake Mix
- pattern: ^[hH]1[aA]
label: Age-1 Hake
- pattern: ^[hH][mM]
label: Hake Mix
- pattern: hake_mix
label: Hake Mix
- pattern: hake
label: Hake
HAUL_NUM:
- pattern: '[0-9]+'
label: None
COUNTRY:
- pattern: ^[cC]
label: CAN
- pattern: ^[uU]
label: US
pattern: "{REGION_CLASS}{HAUL_NUM}{COUNTRY}"
inpfc_strata_region:
CAN: [6]
US: [1, 2, 3, 4, 5]
save_file_template: "{COUNTRY}_{YEAR}_transect_region_haul_{GROUP}.xlsx"
save_file_directory: /Stratification
save_file_sheetname: Sheet1


#####################################################################################################################
# Log-linear regression#
########################
# Target strength (TS) - length (L) regression: TS=m*log10(L)+b
TS_length_regression_parameters:
pacific_hake: # corresponding species text code
number_code: 22500 # species number code
TS_L_slope: 20.0 # the 'm' or 'slope' parameter
TS_L_intercept: -68.0 # the 'b' or 'y-intercept'
length_units: cm # units for L used in regression/relationship

#####################################################################################################################
# Geospatial settings#
########################
geospatial:
init: epsg:4326 # EPSG integer code for geodetic parameter dataset

#####################################################################################################################
# Kriging parameters#
#####################
kriging_parameters:
A0: 6.25 # base area of the grid cell
longitude_reference: -124.78338 # longitude reference for kriging mesh adjustment
longitude_offset: -124.78338 # longitudinal offset for kriging mesh adjustment
latitude_offset: 45.0 # latitudinal offset for kriging mesh adjustment

#####################################################################################################################
...
107 changes: 107 additions & 0 deletions config_files/initialization_config_2013.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# This YAML file is a configuration file for all
# initialization parameters used in Echopop

---
#####################################################################################################################
# Survey data processing#
########################
# Jolly and Hampton (1990) stratified mean calculation
stratified_survey_mean_parameters:
strata_transect_proportion: 0.75 # percent of Jolly-Hampton transects in each stratum
num_replicates: 10000 # number of resample replicates
mesh_transects_per_latitude: 5 # number of virtual transects per degree latitude in kriging mesh

#####################################################################################################################
# Biological data processing#
########################
# Jolly and Hampton (1990) stratified mean calculation
bio_hake_len_bin: [ # length sequence array 2 - 80 cm. This specifies the bin centers!
2, # start of bin centers
80, # end of bin centers
40 # number of bins in total
]

bio_hake_age_bin: [ # age sequence array: year - age 1-22
1, # start of sequence
22, # end of sequence
22 # number of points in the sequence
]

#####################################################################################################################
# Acoustic transect export processing#
########################
nasc_exports:
export_file_directory: /raw_nasc
nasc_export_directory: /Exports # Acoustic export directory
save_file_template: "{REGION}_NASC_{YEAR}_table_{GROUP}.xlsx"
save_file_sheetname: Sheet1
regions: # Main acoustic data region names (list or single string)
all_ages: [Age-1 Hake, # Age-1+
Age-1 Hake Mix,
Hake,
Hake Mix]
no_age1: [Hake, Hake Mix] # Age-2+
max_transect_spacing: 10.0 # Maximum transect spacing (nmi)
file_columns: [transect_num, region_id, distance_s, distance_e, latitude, longitude,
stratum_num, transect_spacing, layer_mean_depth, layer_height, bottom_depth,
NASC, haul_num]
transect_region_mapping:
parts:
REGION_CLASS:
- pattern: ^[hH](?![a-zA-Z]|1a)
label: Hake
- pattern: ^[hH]1[aA][mM]
label: Age-1 Hake Mix
- pattern: ^[hH]1[aA]
label: Age-1 Hake
- pattern: ^[hH][mM]
label: Hake Mix
- pattern: hake_mix
label: Hake Mix
- pattern: hake
label: Hake
HAUL_NUM:
- pattern: '[0-9]+'
label: None
COUNTRY:
- pattern: ^[cC]
label: CAN
- pattern: ^[uU]
label: US
pattern: "{REGION_CLASS}{HAUL_NUM}{COUNTRY}"
inpfc_strata_region:
CAN: [6]
US: [1, 2, 3, 4, 5]
save_file_template: "{COUNTRY}_{YEAR}_transect_region_haul_{GROUP}.xlsx"
save_file_directory: /Stratification
save_file_sheetname: Sheet1


#####################################################################################################################
# Log-linear regression#
########################
# Target strength (TS) - length (L) regression: TS=m*log10(L)+b
TS_length_regression_parameters:
pacific_hake: # corresponding species text code
number_code: 22500 # species number code
TS_L_slope: 20.0 # the 'm' or 'slope' parameter
TS_L_intercept: -68.0 # the 'b' or 'y-intercept'
length_units: cm # units for L used in regression/relationship

#####################################################################################################################
# Geospatial settings#
########################
geospatial:
init: epsg:4326 # EPSG integer code for geodetic parameter dataset

#####################################################################################################################
# Kriging parameters#
#####################
kriging_parameters:
A0: 6.25 # base area of the grid cell
longitude_reference: -124.78338 # longitude reference for kriging mesh adjustment
longitude_offset: -124.78338 # longitudinal offset for kriging mesh adjustment
latitude_offset: 45.0 # latitudinal offset for kriging mesh adjustment

#####################################################################################################################
...
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
Hake Mix]
no_age1: [Hake, Hake Mix] # Age-2+
max_transect_spacing: 10.0 # Maximum transect spacing (nmi)
file_columns: [transect_num, region_id, vessel_log_start, vessel_log_end, latitude, longitude,
file_columns: [transect_num, region_id, distance_s, distance_e, latitude, longitude,
stratum_num, transect_spacing, layer_mean_depth, layer_height, bottom_depth,
NASC, haul_num]
transect_region_mapping:
Expand Down
Loading
Loading