Enable Continuous Integration Testing

[rburghol]

The goal is to develop a set of scripts to verify performance and execution of all facets of the hsp2 install upon making changes, so that we can easily see if anything obvious has broken as we improve things. Thus far, testing is being setup on the forked project at https://github.com/HARPgroup/HSPsquared/, and once we arrive at a working setup we will share it back here.

Tasks

• Test setup CI on forked repo with install of target environment and packages
• Create simple pytest script for proof of concept
• Create hsp2-install-test.yml file to house base testing (over environment.yml config)
  • Develop pip installer test action https://github.com/HARPgroup/HSPsquared/blob/testx/.github/workflows/hsp2-pip-install-test.yml
• Create hsp2-install-test-dev.yml file to house base testing (over environment_dev.yml config)
  • Develop pip installer test action https://github.com/HARPgroup/HSPsquared/blob/testx/.github/workflows/hsp2-pip-install-test-dev.yml
  • Develop conda installer test action (see Conda below)
• Enable running test10 in hsp2-install-test.yml
  • Create the h5 and import the UCI
  • Create pytest_test10.py to do summaries assert that h5 file is created
• Lay out basic parameters for cost-efficient testing (per https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions and https://docs.github.com/en/billing/managing-billing-for-github-packages/about-billing-for-github-packages)
• Create pull request to develop Testing framework base deployment #142

Conda

• github VMs supposedly have an environment variable CONDA set up, so:
  • install with $CONDA/bin/conda env update --file environment.yml --name base
  • see: https://autobencoder.com/2020-08-24-conda-actions/

Notes

• Based loosely on https://coderefinery.github.io/testing/continuous-integration/, and then during testing I have updated to reflect some suggestions/variations from https://docs.github.com/en/enterprise-server@3.11/actions/automating-builds-and-tests/building-and-testing-python
  • Also:
    • On windows to test pytest with pip, run pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --default-timeout=1000 pytest pytest-cov
• Base enabling complete, now working through error when pytest runs the file tests/convert/conversion_test.py.
  • pytest has automatic discovery (if run without explicit arguments) https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#test-discovery

Start at Actions page:

Disabling a Test

Enabling a Test

Cost Factors for Testing

• Basic accounts get 2,000 testing minutes per month free from github (paid accounts get 3,000)
• Current pip/Linux test path takes ~4.1 mins to execute
  • Uses test10
  • Install/dependencies 22 secs
  • Install hsp2 37 secs
  • Import+Run hsp2 ~3.0 mins
• Current conda/Linux test path takes N/A to execute
  • Uses test10
  • Install/dependencies 1.3 mins
  • Install hsp2 1 secs (this seems rather short, unless the conda install took care of all dependencies)
  • Import+Run hsp2 failed
• Loading estimates:
  • Note: test triggering is by branch, and is off by default, i.e., you have to specify a branch as a target of a test.
  • Tests can be triggered on PRs or pushs (commits that get pushed to github from someones local machine)
  • Last month there were 7 pull requests on develop, and 27 commits on develop-specact
  • On our fork repo we 95 commits, more than 70 of which were tiny incremental commits done to test the testing framework, so, a reasonable max monthly estimate might be 50 commits per month.
  • If testing were enabled for:
  • all PRs to develop and master,
  • all commits to the develop-specact:
  • all testing was performed on respec (which is unlikely),
  • 740 minutes / month:
    • ~ 90 test events per month
    • 4 mins per test * 2 tests (normal and dev install)
    • ~740 minutes / month

@PaulDudaRESPEC

[PaulDudaRESPEC]

Setting up Continuous Integration would be a really nice step forward for the project. My first suggestion would be to work with test10 (tests/test10), as it includes a full suite of hydrology, sediment, and water quality.

[rburghol]

I am running into a snag on the CI setup with it failing at the "lint" step, on the file setup.py. The error is:

./setup.py:87:13: F821 undefined name '__version__'

I can see that the file _version.py establishes the __version__ variable, and that file appears to be loaded in setup.py line 8 (with exec(open("./_version.py").read())), so I'm unsure as to why it appears to causing a problem in the lint testing. I know that we have made some small tweaks to setup.py over the last 12 months, anyhow, if anyone has any ideas I am all for it. Hey @timcera I think you set this up originally, do you have any thoughts you could throw out there?

Note: we are testing this CI in github over on a forked repo at https://github.com/HARPgroup/HSPsquared

[TimothyCera-NOAA]

flake8 doesn't run the code, so the "exec(open("./_version.py").read())" isn't actually executed to set the version variable. You have to setup flake8 to ignore it.

[rburghol]

Oh sweet - worked like a charm - thanks @TimothyCera-NOAA !!

[rburghol]

• numba 0.53.1 is unavailable on the gihub Ubuntu latest test env

[timcera]

It isn't Ubuntu. numba 0.53.1 isn't available for Python 3.10. https://pypi.org/project/numba/0.53.1/#files

[timcera]

You need 0.55.0 or later for Python 3.10 support https://pypi.org/project/numba/0.55.0/#files

[timcera]

Numba's latest version (0.59.*) dropped support for Python 3.8, which is HSPSquared's target Python. It seems Numba supports three Python versions back from the latest Python version.

Numba 0.56.0 Python 3.7 to 3.10
Numba 0.57.0 Python 3.8 to 3.11
Numba 0.58.0 Python 3.8 to 3.11
Numba 0.59.0 Python 3.9 to 3.12

[rburghol]

Thanks a ton @timcera, that makes sense. I noticed that the build system did not autodetect the correct python version (choosing 3.10), so I will dive in there.

I know there is some support around here for moving forward to newer versions of python+numba, and then the complication when increasing numba version is that tables 3.6.1 breaks numba -- at least with the default 0.59.

I think what I will do is to create separate test setups to evaluate the current working setup (py 3.8), and the do a separate one for forward looking installs.

[aufdenkampe]

@rburghol, @PaulDudaRESPEC, and @timcera.

Thanks for moving all this forward with recent releases and the move to CI/CD.

I'm a big fan of keeping the dev/target environment up to date. Moving to Python 3.10 or even 3.11 is a really good idea. One approach to figuring out what versions are broadly supported is to follow the latest Anaconda meta-package releases: https://docs.anaconda.com/free/anaconda/pkg-docs/

Anaconda does substantial testing for cross compatibility of most scientific Python dependencies, so once Anaconda supports a new version of Python, it's generally safe to adopt. So, Anaconda's default configuration has used Python 3.10 since March 2023 and started supporting Python 3.11 around that time.

When we were actively developing v0.9 and v0.10 in Fall of 2021, there were a number of very quirky issues and incompatibilities with different versions of numpy, numba, and pyTables, so we had to pin to very specific versions. Those issues have all cleared out.

Note that for the v0.10 release, we were starting to use environment_dev.yml, with Python 3.9 and pyTables 3.7 and other updates, and it was working quite well.

The approach I would take to updating would be to modify environment_dev.yml by updating to python 3.10 and unpinning all versions of numpy, numba, pyTables, and HDF5. It's probably also worth changing to pandas <2 to get the latest version of the pandas 1.x series. Then let conda/mamba solve for the environment. If you use conda, I recommend using --solver=libmamba flag to speed up the process.

[aufdenkampe]

I just added and tested some potential updates to the conda environment files in the new dependency_options branch with commits 00748d8 and 4dfc4f3. Both environments solved quickly with libmamba.

See:

• https://github.com/respec/HSPsquared/blob/dependency_options/environment.yml
• https://github.com/respec/HSPsquared/blob/dependency_options/environment_dev.yml