Skip to content

Conversation

@suvarchal
Copy link
Collaborator

@suvarchal suvarchal commented Jun 24, 2025

adds better test automation framework then our custom automation we have to test fesom.

  • mainly support for agentic development, just build and run standard ctest to see if changes worked.
  • uses standard ctest framework and dir name: tests
  • test data follows directory structure similar to levante
  • can easily test matrix tests with different initial data, forcings, and meshes.
  • currently only integration tests, but leaves more options for future.
  • this will be optional tests for now, eventually we can get rid of current test dir till that switch is made we keep both.

@suvarchal suvarchal marked this pull request as draft June 24, 2025 23:20
@suvarchal suvarchal marked this pull request as ready for review July 2, 2025 09:48
@sebastianbeyer
Copy link
Collaborator

Pretty cool! I did not yet find the integration tests themselves, but I will keep looking xD
Do you already cover all of the tests that we currently have? Once that is true, we can remove the current github tests, correct?

@suvarchal
Copy link
Collaborator Author

suvarchal commented Jul 3, 2025

@sebastianbeyer https://github.com/FESOM/fesom2/blob/add_ctest_automation/cmake/FesomTesting.cmake is where the integration tests are, it is meant inside cmake dir because to use these as sort of fixtures in future for all other tests for different setups. But right now only main fesom test is done similar to what happens in fesom general test in CI. May restructure this once more tests are added. this was trying to have minimal changes for PR.

to test:

./configure.sh ubuntu -DBUILD_TESTING=ON 
cd build
ctest -V

there is more model changes to be done before all tests are work as intended but this is start, right now model exits with exit code 0 for even namelist errors. which any testing says it passed, we are forced to check logs and/or output to know for sure we were successful. Insted we should use exit codes so that we can cleanly know if run was successful (will roll that out soon in one of next PRs).

@JanStreffing JanStreffing merged commit 71a2112 into main Jul 4, 2025
8 checks passed
@JanStreffing JanStreffing deleted the add_ctest_automation branch July 4, 2025 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants