Add population-level calculations and stratified statistics

[brandynlucca]

Several functions were added to calculate the age- and sex-stratified areal number/biomass densities, summed abundance, and summed biomass estimates. This includes the following changes:

• initialization_config.yml: added a field for replicates specific to the stratified resampling procedure.
• spatial.py
  • calculate_bounds: calculates the latitude/longitude boundary box around each transect line
  • calculate_transect_distance: calculates the length distance and area of each transect line necessary for the stratified statistic calculation
• statistics.py
  • confidence_interval: calculates the 95% confidence interval (for a Normal distribution)
  • stratified_transect_statistic: produces estimates and confidence intervals (around the mean) for the following statistics/estimators: mean, variance, coefficient of variation (CV)
• survey.py
  • nasc_to_biomass_conversion: converts NASC and biological data into areal biomass/number density, summed biomass, and summed abundance stratified by age, sex, stratum, and transect
    • Nested within transect_analysis (pre-existing Survey method)
  • stratified_summary: wrapper function for stratified_transect_statistic and calculate_transect_distance that adds mean and confidence interval estimates for the stratified mean, variance, and coefficient of variation (CV) produced by the resampling/bootstrapping procedure
    • New Survey method

Current use

from EchoPro.survey import Survey

# Load configuration settings and all data
obj = Survey( init_config_path='./config_files/initialization_config.yml' , survey_year_config_path='./config_files/survey_year_2019_config.yml' )

# Calculate base transect-based results necessary for later calculations
obj.transect_analysis()

# Calculate transect-based stratified results
obj.stratified_summary()

# Calculate biomass 
print(f"Total adult biomass: {1e-6*self.biology[ 'population' ][ 'biomass' ][ 'biomass_df' ].pipe( lambda df: df.loc[ ( df[ 'sex' ] == 'total' ) ] ).B_adult.sum():.3f} kmt")

Total adult biomass: 1655.225 kmt

print(f"Adult biomass CV: {self.biology[ 'population' ][ 'stratified_results' ][ 'biomass' ][ 'CV' ][ 'estimate' ]*1e2:.2f}% [{ci_bounds[0]*1e2:.2f}%, {ci_bounds[1]*1e2:.2f}%; 95 pct. CI]" )

Adult biomass CV: 13.32% [12.40%, 14.23%; 95 pct. CI]

[brandynlucca]

Commit 1cdcf4d was also pushed to see how how the CI testing handled the change. The change in load_configuration from a class method to a standalone function results in an expected failed test state. This makes sense given that L22-25 within test_data_loader.py still expects load_configuration to be a Survey class method.

I am just noting this as a reminder to myself to amend the test_data_loader.py accordingly and have created Issue #177 to track both this failed test and future bugs/errors for the integration tests as a whole.

[leewujung]

@brandynlucca : Thanks for the PR, nice work!

My inline comments are small. Below are some high-level organization comments, regarding the function nasc_to_biomass_conversion.

Some observations

1. L862-L937, the code does 2 things:
   • prepare dataframes_to_add, which is used to calculate nasc_fraction_total_df
   • prepare sex_indexed_weight_proportions, which is used to calculate age_biomass_df
2. L939-L950:
   • calculate nasc_fraction_total_df
3. L939-L1022:
   • everything is derived from nasc_to_areal_number_density_df (and other stored data within the Survey object)
4. L1032-1038:
   • age_biomass_df is calculated
5. At the end the key dataframes are saved

Suggestions

Since this function is very long, I suggest factoring out the section (1. and 2. above) into two separate functions, one that produces nasc_fraction_total_df and the other produces sex_indexed_weight_proportions. These two functions would be basically helper function that prepares data.

This way the main flow of your calculation will be very clearer, since people who look into the code do not have to keep track of the large number of dfs from the beginning, since they are merged/grouped nicely into columns with good names in nasc_fraction_total_df and sex_indexed_weight_proportions.

The separation will also help with writing tests, so that if something fails, the chunk to isolate to figure out potential bugs is smaller.

[brandynlucca]

@leewujung I think I have addressed everything you've mentioned so far. I am now in the process of stepping backward to enable the age-1 inclusion/exclusion flag. I can include that in a separate PR while also constructing more basic testing functions.

[leewujung]

@brandynlucca : Could you fix test breaking? Seems it's a missing import. Thanks.

[leewujung]

I am now in the process of stepping backward to enable the age-1 inclusion/exclusion flag. I can include that in a separate PR while also constructing more basic testing functions.

I would suggest just having the tests of what you have in a single PR.

Let's do the age-1 inclusion/exclusion later, after kriging and reports/figures, so we have the entire process completed and verified first.

[leewujung]

@brandynlucca : Thanks for the changes. I realized that I did not review the function stratified_transect_statistic, so I just read through it now and only had a few minor suggestions and questions about NaN values.

The main higher-level question I have is on enabling stratified_transect_statistic to use
both INPFC and KS stratum. It seems that the only place needing change (ie adding an if-else case) would be the resampling part that needs to choose from a list from that "stratification1" sheet (I forgot what you called it in the dataframes) rather than from this np.range array.

If there is not already an issue tracking this, we can add one and add this as one of the TODO items.

Otherwise I think this PR is ready to be merged once the small changes are made. :)

Also, parking some thoughts here regarding documentation: I think we can include the equations implemented here into the docs with references, so that we could stop the confusion or ambiguity once and for all. Let's discuss this more once the code is more settled.