Migrate timemap/refcase dependent observations

[yngve-sk]

(Goes with equinor/semeio#862)

Overview of changes

• Remove REFCASE, TIME_MAP, and HISTORY_OBSERVATION from ERT
• Add tool to migrate older ERT configs so that information from REFCASE and TIME_MAP is embedded into the obs config
• GENERAL_OBSERVATION can now only be used with RESTART, usage of DATE, HOURS, DAYS can be migrated if there is a REFCASE or TIME_MAP in the ERT config.
• SUMMARY_OBSERVATION can now only be used with DATE, usage of RESTART, HOURS, DAYS can be migrated if there is a REFCASE or TIME_MAP in the ERT config.

Nice to know:

• Tests that were previously testing now deprecated behaviors have been kept by shimming the obs config migration into the test.
• Removed test case and logic for asserting that a SUMMARY_OBSERVATION's date matches up with the refcase / time map (this will be hard to pick up in migration, so suggest just dropping it)

Issue
Resolves #12619

Residual (IDEALLY NON BLOCKING) issues/TODOs
Recommend resolving before next release:

• Remove FOPR observations from snake oil (i.e., remove the whole set of observations, and expect some snapshot changes) Remove some or all FOPR observations from snake oil #12681
• Make _GeneralObservation.restart non optional Make _GeneralObservation.restart non optional #12682
• Reconsider the precision of the history obs Migrate timemap/refcase dependent observations #12629 (comment) Change precision of migrated history observation values #12683

Should def be fixed:

• Move validations/errors wrt observations parsing from _create_observation_dataframes.py to _observations.py. Will likely fix this in the following PR where *Observations are converted to BaseModel objs, opening up for moving validations there (instead of doing the the dataclasses.dataclass way now) Move validations/errors wrt observations parsing #12680

Not urgent:

• Don't write info about the CLI tool into _observations.py, ref Migrate timemap/refcase dependent observations #12629 (comment) . Handle observation errors and migrations higher in the stack #12685
• Port the poly template example to not use history obs, right now it creates a refcase and uses history obs to make observations based on that. Current workaround is just migrating it Rewrite poly template to stop using Summary observations with restart #12684
• Replace test usages of run_convert_observations with just migrating the observations and taking it from there (more instances of this can be found) Replace test usages of run_convert_observations #12686

[Copilot]

Pull request overview

This PR introduces a migration tool to convert deprecated HISTORY_OBSERVATION to SUMMARY_OBSERVATION and remove dependencies on TIME_MAP and REFCASE from observation configurations. The tool aims to help users migrate away from deprecated features as part of issue #12619.

Changes:

• Adds new ert_config_migrations.py module with observation migration logic
• Adds convert_observations CLI command to __main__.py

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 18 comments.

File	Description
src/ert/config/ert_config_migrations.py	New module implementing observation migration logic including text editing, observation conversion, and configuration parsing
src/ert/main.py	Adds CLI subcommand convert_observations to invoke the migration tool

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.63519% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.63%. Comparing base (7238d60) to head (029718f).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/ert/config/observation_config_migrations.py	94.08%	23 Missing ⚠️
src/ert/config/_observations.py	93.10%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12629      +/-   ##
==========================================
- Coverage   90.66%   90.63%   -0.04%     
==========================================
  Files         430      431       +1     
  Lines       29814    30067     +253     
==========================================
+ Hits        27032    27252     +220     
- Misses       2782     2815      +33     

Flag	Coverage Δ
cli-tests	37.33% <30.90%> (-0.27%)	⬇️
gui-tests	68.72% <32.61%> (-0.63%)	⬇️
performance-and-unit-tests	74.05% <92.91%> (+0.10%)	⬆️
test	37.57% <3.21%> (-0.49%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[codspeed-hq]

Merging this PR will not alter performance

✅ 22 untouched benchmarks

---

_{Comparing yngve-sk:26.01.remove-history-obs-with-utility (029718f) with main (7238d60)}

[oyvindeide]

Can do if that is most ideal? Was not sure whether we should do an inline edit to the ert config and not only the obs config. In the case that someone wants to add a new history obs, and then migrate it, removal of time map / refcase would block that.

Think that should be covered by tests, and it should be removed from snake_oil as that no longer has HISTORY_OBSERVATION.

[berland]

I say let's go!

[oyvindeide]

Nice! Just some minor comments

[oyvindeide]

Minor comment, but the KEY should be GOPR:FIELD and the name of the observation something else.

[oyvindeide]

How much work is it to raise this as a specific error that we then catch higher up to add context, for example SummaryObservationError, the reason being that we dont need the context of the cli tool in this part of the code, that can live higher in the stack. Should still be available in the suggester in the same way as now, so should reraise it as ObservationConfigError higher up (or another type that brings it into the suggestor)

[yngve-sk]

ObservationConfigError is a ConfigValidationError, so it should show up in suggester?

[oyvindeide]

The message is good, I was thinking about the fact that _observations.py knows there exists and cli tool, while logically that information should exist higher in the stack _observations.py only knows that HISTORY_OBSERVATION is deprecated, and we can add more context higher in the stack.

[yngve-sk]

Oh, I see. I think this could be a residual followup issue, likely requires some design decisions (first idea that comes to my mind is having an own observation error type, indicating that it needs migration, and taking it from there) and moving of some things.

[andreas-el]

This will also allow us to inject the path to the ert-file, so that the feedback message actually contains a valid command.
E.g.
ert convert_observations test-data/ert/poly_example/poly.ert instead of that placeholder.