KeyError: "['stratum'] not in index" error in load_survey_data()

[emiliom]

I ran into an error on my first attempt at running EchoPro using the minimal_echopro_workflow.ipynb notebook. Specifically, on cell 3, survey_2019.load_survey_data(), I get this error:

KeyError: "['stratum'] not in index"

@b-reyes could you look into it?

I'm using the same set of input files I used in July.

I installed the echopro environment using a slight variation of your instructions and requirements file, but my changes are specifically focused on jupyter/ipython packages so they should not affect anything being done in survey_2019.load_survey_data(). Specifically:

mamba create -c conda-forge -n echopro python=3.9 ipykernel --file requirements.txt
conda activate echopro
pip install -e .

Where requirements.txt is:

geopandas==0.10.2
PyYAML==6.0
xarray==2022.3.0
geopy==2.2.0
openpyxl==3.0.9
ipywidgets==7.7.0
ipympl==0.8.7
numba==0.55.1
# notebook==6.4.11
# nb_conda_kernels==2.3.1
# jupyter_contrib_nbextensions==0.5.1

Here is the complete error trace:

KeyError                                  Traceback (most recent call last)
File <timed eval>:1

File /usr/mayorgadat/workmain/acoustics/gh/uw-echospace/EchoPro/EchoPro/survey/survey.py:222, in Survey.load_survey_data(self, file_type)
    220 # load all associated stratification data
    221 if file_type in ('strata', 'all'):
--> 222     LoadStrataData(self)
    224 if file_type in ('nasc', 'all'):
    225     self.nasc_df = load_nasc_data.load_nasc_df(self)

File /usr/mayorgadat/workmain/acoustics/gh/uw-echospace/EchoPro/EchoPro/load_stratification_data/load_stratification_data.py:29, in LoadStrataData.__init__(self, survey)
     26 # expected columns for geo strata Dataframe
     27 self.geo_strata_cols = {'Strata index', 'Latitude (upper limit)'}
---> 29 self._load_stratification_file()
     30 self._load_geographic_stratification()
     31 self._get_strata_sig_b()

File /usr/mayorgadat/workmain/acoustics/gh/uw-echospace/EchoPro/EchoPro/load_stratification_data/load_stratification_data.py:63, in LoadStrataData._load_stratification_file(self)
     60 self._check_strata_df(strata_df)
     62 # extract only those columns that are necessary
---> 63 strata_df = strata_df[['Year', 'stratum', 'Haul', 'wt']].copy()
     65 # set data types of dataframe
     66 strata_df = strata_df.astype({'Year': int, 'stratum': int,
     67                               'Haul': int, 'wt': np.float64})

File ~/miniconda3/envs/echopro/lib/python3.9/site-packages/pandas/core/frame.py:3511, in DataFrame.__getitem__(self, key)
   3509     if is_iterator(key):
   3510         key = list(key)
-> 3511     indexer = self.columns._get_indexer_strict(key, "columns")[1]
   3513 # take() does not accept boolean indexers
   3514 if getattr(indexer, "dtype", None) == bool:

File ~/miniconda3/envs/echopro/lib/python3.9/site-packages/pandas/core/indexes/base.py:5796, in Index._get_indexer_strict(self, key, axis_name)
   5793 else:
   5794     keyarr, indexer, new_indexer = self._reindex_non_unique(keyarr)
-> 5796 self._raise_if_missing(keyarr, indexer, axis_name)
   5798 keyarr = self.take(indexer)
   5799 if isinstance(key, Index):
   5800     # GH 42790 - Preserve name from an Index

File ~/miniconda3/envs/echopro/lib/python3.9/site-packages/pandas/core/indexes/base.py:5859, in Index._raise_if_missing(self, key, indexer, axis_name)
   5856     raise KeyError(f"None of [{key}] are in the [{axis_name}]")
   5858 not_found = list(ensure_index(key)[missing_mask.nonzero()[0]].unique())
-> 5859 raise KeyError(f"{not_found} not in index")

KeyError: "['stratum'] not in index"

[b-reyes]

@emiliom I changed some of the column names to be more inline with the names Alicia and me talked about. I forgot to upload the changed files. They are now uploaded 2019_consolidated_files, sorry about that.

Please keep this issue open though, it appears as if one of my checks should have been triggered and it wasn't. I need to fix this.

[emiliom]

Thanks. The notebook now runs to completion. Great.

One thing to keep in mind for these files downloaded from Google Drive is that the "&" character in file names gets changed to "_". Or at least that happens in my setup, with Firefox on Ubuntu. It happened now, and I remember it had happened before. It must happen when I select the entire folder for download, when GD does on the fly compression to a zip; if I download a single file, the name is not changed. I'm not suggesting we do anything about this at this time. Just making a note of it.

As we discussed, please wait until Milestone 3 to address input file tweaks or input and configuration file checks. They're not priorities at this time. Let's focus on what's in Milestone 2.

[emiliom]

One tweak to what I said about minimal_echopro_workflow.ipynb running to completion: the semivariogram widget generated by semi_vario.get_widget() came out as a simple static plot rather than as an actual widget. I'm not going to pay much attention to that right now, since the changes I made to the installation steps may have impacted ipywidgets. I'll get back to it later this week.

[b-reyes]

One tweak to what I said about minimal_echopro_workflow.ipynb running to completion: the semivariogram widget generated by semi_vario.get_widget() came out as a simple static plot rather than as an actual widget. I'm not going to pay much attention to that right now, since the changes I made to the installation steps may have impacted ipywidgets. I'll get back to it later this week.

@emiliom Perhaps this is because you are using Jupyter Lab? I am not using Jupyter Lab and the result is not a static plot, it is an actual widget.

[b-reyes]

@emiliom I have reviewed the code that was checking the column names. As I suspected, the unexpected behavior that I was referencing above was resolved in the 0.1.0-alpha release. However, the error output was not that informative. To make the error output better, I have created PR #36.

Once PR #36 is merged, I believe we can close this issue.

[emiliom]

I have reviewed the code that was checking the column names. As I suspected, the unexpected behavior that I was referencing #28 (comment) was resolved in the 0.1.0-alpha release.

Actually, strictly speaking I think the error was simply due to the fact that the input file column names had been changed, and the code reflected those names. As I mentioned above, EchoPro ran to completion back then once I used your updated input files.

However, the error output was not that informative. To make the error output better, I have created PR #36.

Great!

[b-reyes]

Actually, strictly speaking I think the error was simply due to the fact that the input file column names had been changed, and the code reflected those names. As I mentioned above, EchoPro ran to completion back then once I used your updated input files.

I was not referring to the error that you were experiencing, rather I was referencing the fact that my check should have been triggered when a column name was not found. Either way, PR #36 fixes this.

[emiliom]

@b-reyes we can close this issue, right?

[b-reyes]

@b-reyes we can close this issue, right?

Yes, we can close this issue. I just merged #36, which addresses the main concerns in this issue. I will go ahead and close this.