Tests: end-to-end verify solver_options_layers reach gurobi at solve time (#700)

* tests: end-to-end verify solver_options_layers reach gurobi at solve time

A new gurobi-only test module
``mpisppy/tests/test_options_reach_solver.py`` exercises every
layer-source path documented in
``doc/designs/solver_options_redesign.md``: each test sets one
option through a specific surface (CLI flag, --solver-options-file
sub-block, per-spoke file, --max-solver-threads cap, deprecated
--mipgaps-json) via the Config / cfg_vanilla flow, runs farmer PH
for a couple of iterations against gurobi_persistent with
``solver_log_dir`` set to a tmpdir, then greps the per-solve
Gurobi log files for the ``Set parameter X to value Y`` line that
proves the parameter actually reached the solver at the expected
iteration.

12 tests covering:
- ``--solver-options`` inline: mipgap, threads (canonical→native
  rename), and a non-translated key (presolve).
- ``--iter0-mipgap`` and ``--iterk-mipgap`` apply at the right
  iteration.
- ``--mipgaps-json`` (deprecated path): per-iteration schedule
  fires at the right N; the N=0 entry is routed through the
  ``default`` predicate per the validator.
- ``--solver-options-file`` per sub-block: ``default`` applies
  everywhere; ``iter0`` and ``iterk`` fire at their predicates
  separately; ``starting_at_iter`` fires from N onward; CLI sugar
  flags override file entries at the same predicate per axis 2.
- Per-spoke layers: global file's ``spokes.<name>`` sub-block AND
  dedicated ``--{name}-solver-options-file`` both reach the
  spoke's solver (exercised through apply_solver_specs so the
  test stays single-process).
- ``--max-solver-threads`` cap wins over any user-supplied
  ``threads`` value.

The test class is decorated with ``skipUnless(gurobi_persistent
available)`` so it cleanly skips when gurobi isn't installed. The
CI gurobi-via-pip job runs it; run_coverage.bash also runs it
locally when available.

## Companion code fix in spopt.solve_one

Setting ``s._solver_plugin.options["LogFile"]`` must happen
BEFORE the solver-options loop. With GurobiPersistent, the model
instance is reused across iterations, and setParam writes
``Set parameter X to value Y`` to the *currently active* LogFile.
With the previous order (options first, then LogFile), each
iteration's parameter-set lines landed in the previous
iteration's still-open log, mis-attributing per-iteration options
to the wrong log file. Reordering puts the LogFile change first
so parameter-set lines land in the right per-solve log. No
behavior change for the actual solves; only the log-file
attribution.

## CI wiring

Added under the existing ``straight-tests-gurobi`` job (which is
already gated on the gurobi-via-pip install + the
``Probe gurobi_persistent availability`` step). The new test file
runs as its own pytest step after straight_tests.py and uploads
into the same ``coverage-straight-tests-gurobi`` artifact, so
codecov picks it up.

Local: 12/12 pass in this file; 237/238 pass in the wider sweep
(unrelated pre-existing test_proper_bundler failure on main).
ruff clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ci: install pytest in straight-tests-gurobi job

The new `Options-reach-solver end-to-end tests` step invokes
pytest, which the gurobi-via-pip job didn't pip-install. Adding
pytest to the install line fixes the `No module named 'pytest'`
error.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: DLWoodruff

.github/workflows/test_pr_and_main.yml

@@ -436,7 +436,7 @@ jobs:
       - name: Install dependencies
         run: |
           conda install mpi4py pandas setuptools
-          pip install pyomo gurobipy dill matplotlib scipy coverage
+          pip install pyomo gurobipy dill matplotlib scipy coverage pytest
 
       - name: setup the program
         run: |
@@ -452,6 +452,10 @@ jobs:
           cd mpisppy/tests
           coverage run $COV_ARGS straight_tests.py --python-args="$PYARGS"
 
+      - name: Options-reach-solver end-to-end tests
+        run: |
+          coverage run $COV_ARGS -m pytest mpisppy/tests/test_options_reach_solver.py -v
+
       - name: Upload coverage data
         if: always()
         uses: actions/upload-artifact@v4

mpisppy/spopt.py

@@ -180,6 +180,30 @@ def _vb(msg):
             results = self.extobject.pre_solve(s)
 
         solve_start_time = time.time()
+
+        # Switch the solver's log destination BEFORE writing other
+        # solver options. For persistent Gurobi, setParam writes a
+        # "Set parameter X to value Y" line at parameter-set time to
+        # whatever LogFile is currently active. If we set MIPGap and
+        # other options first and then redirect LogFile, the
+        # parameter-set lines for THIS iteration land in the PREVIOUS
+        # iteration's still-open log file — which mis-attributes
+        # parameters to the wrong iteration and breaks log-driven
+        # debugging of per-iteration options.
+        solve_keyword_args = dict()
+        if self.options.get("solver_log_dir", None):
+            if k not in self._subproblem_solve_index:
+                self._subproblem_solve_index[k] = 0
+            dir_name = self.options["solver_log_dir"]
+            file_name = f"{self._get_cylinder_name()}_{k}_{self._subproblem_solve_index[k]}.log"
+            # Workaround for Pyomo/pyomo#3589: Setting 'keepfiles' to True is required
+            # for proper functionality when using the GurobiDirect / GurobiPersistent solver.
+            if isinstance(s._solver_plugin, GurobiDirect):
+                s._solver_plugin.options["LogFile"] = os.path.join(dir_name, file_name)
+            else:
+                solve_keyword_args["logfile"] = os.path.join(dir_name, file_name)
+            self._subproblem_solve_index[k] += 1
+
         if (solver_options):
             # Translate canonical option keys to the solver's native
             # spelling at the latest moment, so stored options on
@@ -191,7 +215,6 @@ def _vb(msg):
             for option_key,option_value in translated_options.items():
                 s._solver_plugin.options[option_key] = option_value
 
-        solve_keyword_args = dict()
         if self.cylinder_rank == 0:
             if tee is not None and tee is True:
                 solve_keyword_args["tee"] = True
@@ -210,19 +233,6 @@ def _vb(msg):
             # solve_keyword_args["use_signal_handling"] = False
             pass
 
-        if self.options.get("solver_log_dir", None):
-            if k not in self._subproblem_solve_index:
-                self._subproblem_solve_index[k] = 0
-            dir_name = self.options["solver_log_dir"]
-            file_name = f"{self._get_cylinder_name()}_{k}_{self._subproblem_solve_index[k]}.log"
-            # Workaround for Pyomo/pyomo#3589: Setting 'keepfiles' to True is required
-            # for proper functionality when using the GurobiDirect / GurobiPersistent solver.
-            if isinstance(s._solver_plugin, GurobiDirect):
-                s._solver_plugin.options["LogFile"] = os.path.join(dir_name, file_name)
-            else:
-                solve_keyword_args["logfile"] = os.path.join(dir_name, file_name)
-            self._subproblem_solve_index[k] += 1
-
         Ag = getattr(self, "Ag", None)  # agnostic
         if Ag is not None:
             assert not disable_pyomo_signal_handling, "Not thinking about agnostic APH yet"