Revert one window

.github/workflows/test_pr_and_main.yml

@@ -84,10 +84,6 @@ jobs:
           cd examples
           python afew.py xpress_persistent
 
-      - name: Test ComponentMap
-        run: |
-          pytest mpisppy/tests/test_component_map_usage.py
-
       - name: Test docs
         run: |
           cd ./doc/
@@ -423,6 +419,11 @@ jobs:
         run: |
           pip install -e .
 
+      - name: Build Pyomo extensions
+        run: |
+          # some failures are expected, but this should succeed as long as pynumero is built correctly
+          pyomo build-extensions || python -c "from pyomo.contrib.pynumero.asl import AmplInterface; exit(0) if AmplInterface.available() else exit(1)"
+
       - name: run farmer tests
         timeout-minutes: 10
         run: |

doc/src/amalgamator.rst

@@ -3,11 +3,7 @@
 Amalgamator
 ===========
 
-.. Note::
-   This is an advanced topic, probably of interest only to maintainers of
-   confidence interfal and related code.
-
-The file ``amalgamator.py``
+For simple problems that do not need extra specification, ``amalgamator.py``
 provides several drivers to solve a problem without writing a program
 that creates the cylinders one by one. ``amalgamator.from_module`` enables
 a high-level user to create a hub-and-spoke architecture using the command 
@@ -53,9 +49,7 @@ It must contains the following attributes for use with cylinders:
 
 
 Create Amalgamator from a module and command line
--------------------------------------------------
-
-Given an options
+------------------------------------------------- Given an options
 ``Config`` object (typically called `cfg`) as above,
 ``amalgamator.Amalgamator_parser`` creates calls the appropriate
 functions to add the necessary information for different modules.
@@ -94,7 +88,7 @@ Examples
 --------
 
 As intended, the examples of use of Amalgamator are quite short. The first
-example, ``examples.farmer.archive.farmer_ama.py``, solves directly the EF. The model can be specified,
+example, ``farmer_ama.py``, solves directly the EF. The model can be specified,
 e.g. by taking an integer version of it. This specification can be made via
 the command line, thanks to the ``inparser_adder`` method of ``farmer.py``.
 

doc/src/confidence_intervals.rst

@@ -57,7 +57,7 @@ This object has a ``run`` method that returns a gap estimator and a confidence i
 Examples
 --------
 
-There are example scripts for sequential sampling in both ``farmer/CI`` and ``aircond``.
+There are example scripts for sequential sampling in both ``farmer`` and ``aircond``.
 
 Using stand-alone ``mmw_conf.py``
 ---------------------------------
@@ -79,7 +79,7 @@ to be able to pass problem-specific args down without knowing what they are.
 
 Once a model satisfies the requirement for amalgamator, next a ``.npy`` file should be constructed from the given model. This can be accomplished, for example, by adding the line 
 ``sputils.ef_ROOT_nonants_npy_serializer(instance, 'xhat.npy')`` after solving the ef ``instance``. When using ``Amalgamator`` to solve the program, this can be done by adding the line
-``sputils.ef_ROOT_nonants_npy_serializer(ama_object.ef, "xhat.npy")`` to your existing program (see the example in ``examples/farmer/archive/farmer.py`` for an example of this).
+``sputils.ef_ROOT_nonants_npy_serializer(ama_object.ef, "xhat.npy")`` to your existing program (see the example in ``farmer.py`` for an example of this).
 
 Once this is accomplished, on the command line, run
 ``python -m mpisppy.confidence_intervals.mmw_conf my_model.py xhat.npy gurobi --num-scens n --alpha 0.95``. Note that ``xhat.npy`` is assumed to be in the same directory as ``my_model.py`` in this case. If the file is saved elsewhere then the corresponding path should be called on the command line.

doc/src/examples.rst

@@ -47,7 +47,7 @@ constraints are the number of tons per acre that each crop will yield (2.5 for
 wheat, 3 for corn, and 20 for sugar beets).
 
 
-The following code in ``examples/farmer/archive/farmer.py`` (with similar code in ``examples/farmer/farmer.py``) creates an instance of the farmer's model:
+The following code creates an instance of the farmer's model:
 
 .. testcode::
 
@@ -109,8 +109,7 @@ yields for each crop. We can solve the model:
 The optimal objective value is:
 
 .. testoutput::
-  :options: +SKIP
-
+
     -118600.0
 
 In practice, the farmer does not know the number of tons that each crop will
@@ -207,39 +206,136 @@ farmer's stochastic program.
 Solving the Extensive Form
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The simplest approach is to solve the extensive form of the model directly. Assuming you are in the directory
-``examples/farmer`` the following unix command will work.
+The simplest approach is to solve the extensive form of the model directly.
+MPI-SPPy makes this quite simple:
+
+.. testcode::
 
-.. code-block:: bash
+    from mpisppy.opt.ef import ExtensiveForm
 
-    python ../../mpisppy/generic_cylinders.py --module-name farmer --num-scens 3 --EF --EF-solver-name gurobi
+    options = {"solver": "cplex_direct"}
+    all_scenario_names = ["good", "average", "bad"]
+    ef = ExtensiveForm(options, all_scenario_names, scenario_creator)
+    results = ef.solve_extensive_form()
 
-We can extract the optimal solution itself using the ``--solution-base-name`` option:
+    objval = ef.get_objective_value()
+    print(f"{objval:.1f}")
 
-.. code-block:: bash
 
-    python ../../mpisppy/generic_cylinders.py --module-name farmer --num-scens 3 --EF --EF-solver-name gurobi --solution-base-name farmersol
+.. testoutput::
+
+    ...
+    -108390.0
 
-This command writes solution data for nonanticipative variables to two files with the base name farmersol and full scenario solutions to a directory named farmersol_soldir.
+We can extract the optimal solution itself using the ``get_root_solution``
+method of the ``ExtensiveForm`` object:
+
+.. testcode::
+
+    soln = ef.get_root_solution()
+    for (var_name, var_val) in soln.items():
+        print(var_name, var_val)
+
+.. testoutput::
+
+    X[BEETS] 250.0
+    X[CORN] 80.0
+    X[WHEAT] 170.0
 
-.. note::
-   Most command line options relevant to the EF start with --EF. Most other command line options will be silently ignored
-   if ``--EF`` is specified (one exception is ``--solution-base-name``).
-
 
 Solving Using Progressive Hedging (PH)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Here is a simple command that uses PH as the hub algorithm and
-computes lower bounds using a Lagrangian spoke (``--lagrangian``) with
-upper bounds computed by randomly trying scenario solutions to fix the nonanticipative variables (``--xhatshuffle``).
+We can also solve the model using the progressive hedging (PH) algorithm.
+First, we must construct a PH object:
+
+.. testcode::
+
+    from mpisppy.opt.ph import PH
+
+    options = {
+        "solver_name": "cplex_persistent",
+        "PHIterLimit": 5,
+        "defaultPHrho": 10,
+        "convthresh": 1e-7,
+        "verbose": False,
+        "display_progress": False,
+        "display_timing": False,
+        "iter0_solver_options": dict(),
+        "iterk_solver_options": dict(),
+    }
+    all_scenario_names = ["good", "average", "bad"]
+    ph = PH(
+        options,
+        all_scenario_names,
+        scenario_creator,
+    )
+
+
+.. testoutput::
+    :hide:
+
+    ...
+
+Note that all of the options in the ``options`` dict must be specified in order
+to construct the PH object. Once the PH object is constructed, we can execute
+the algorithm with a call to the ``ph_main`` method:
+
+.. testcode::
+
+    ph.ph_main()
+
+.. testoutput::
+    :hide:
+
+    ...
+
+
+.. testoutput::
+    :options: +SKIP
+
+
+    [    0.00] Start SPBase.__init__
+    [    0.01] Start PHBase.__init__
+    [    0.01] Creating solvers
+    [    0.01] Entering solve loop in PHBase.Iter0
+    [    2.80] Reached user-specified limit=5 on number of PH iterations
 
+Note that precise timing results may differ.  In this toy example, we only
+execute 5 iterations of the algorithm. Although the algorithm does not converge
+completely, we can see that the first-stage variables already exhibit
+relatively good agreement:
 
-.. code-block:: bash
-
-    mpiexec -np 3 python -m mpi4py ../../mpisppy/generic_cylinders.py --module-name farmer --num-scens 3 --solver-name gurobi_persistent --max-iterations 10 --max-solver-threads 4 --default-rho 1 --lagrangian --xhatshuffle --rel-gap 0.01 
+.. testcode::
 
+    variables = ph.gather_var_values_to_rank0()
+    for (scenario_name, variable_name) in variables:
+        variable_value = variables[scenario_name, variable_name]
+        print(scenario_name, variable_name, variable_value)
+
+.. testoutput::
+    :hide:
+
+    ...
+    average X[BEETS]
+    ...
+
+.. testoutput::
+    :options: +SKIP
 
+    good X[BEETS] 280.6489711937925
+    good X[CORN] 85.26131687116064
+    good X[WHEAT] 134.0897119350402
+    average X[BEETS] 283.2796296293019
+    average X[CORN] 80.00000000014425
+    average X[WHEAT] 136.72037037055298
+    bad X[BEETS] 280.64897119379475
+    bad X[CORN] 85.26131687116226
+    bad X[WHEAT] 134.08971193504266
+
+The function ``gather_var_values_to_rank0`` can be used in parallel to collect
+the values of all non-anticipative variables at the root. In this (serial)
+example, it simply returns the values of the first-stage variables.
 
 Solving Using Benders' Decomposition
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

doc/src/extensions.rst

@@ -69,22 +69,12 @@ This is a good extension to look at as a first example. It takes a
 dictionary with iteration numbers and mipgaps as input and changes the
 mipgap at the corresponding iterations. The dictionary is provided in
 the options dictionary in ``["gapperoptions"]["mipgapdict"]``.  There
-is an example of its use in ``examples.sizes.sizes_demo.py``.
-
-Instead of an options dictionary, when run with cylinders the options
-``["gapperoptions"]["starting_mipgap"]`` and ``["gapperoptions"]["mipgap_ratio"]``
-can be set. The ``starting_mipgap`` will be the initial value used,
-and as the cylinders close the relative optimality gap the extension will set the subproblem
-mipgaps as the ``min(starting_mipgap, mipgap_ratio * problem_ratio)``, where
-the ``problem_ratio`` is the relative optimality gap on the overall problem
-as computed by the cylinders.
-
-This extension can also be used with the Lagrangian and subgradient spokes.
+is an example of its use in ``examples.sizes.sizes_demo.py``
 
 fixer.py
 ^^^^^^^^
 
-This extension provides methods for fixing nonanticipative variables (usually integers) for
+This extension provides methods for fixing variables (usually integers) for
 which all scenarios have agreed for some number of iterations. There
 is an example of its use in ``examples.sizes.sizes_demo.py`` also
 in ``examples.sizes.uc_ama.py``. The ``uc_ama`` example illustrates
@@ -99,36 +89,6 @@ to be on the ``Config`` object so the amalgamator can find it.
    So if you don't want to fix a variable at iteration zero, provide a
    tolerance, but set all count values to ``None``.
 
-reduced_cost_fixer
-^^^^^^^^^^^^^^^^^^
-
-This extension provides methods for fixing nonanticipative variables based on their expected
-reduced cost as calculated by the ReducedCostSpoke. The aggressiveness of the
-fixing can be controled through the ``zero_rc_tol`` parameter (reduced costs
-with magnitude below this value will be considered 0 and not eligible for fixing)
-and the ``fix_fraction_target`` paramemters, which set a maximum fraction of
-nonanticipative variables to be fixed based on expected reduced costs. These two
-parameters iteract with each other -- the expected reduced costs are sorted by
-magnitude, and if the `fix_fraction_target`` percental is below ``zero_rc_tol``,
-then fewer than ``fix_fraction_target`` variables will be fixed. Further, to
-have a defined expected reduced cost, all nonant variable values *must be* at
-the same bound in the ReducedCostSpoke.
-
-Variables will be unfixed if they no longer meet the expected reduced cost
-criterion for fixing, e.g., the variable's expected reduced cost became too
-low or the variable was not at its bound in every subproblem in the ReducedCostSpoke.
-
-relaxed_ph_fixer
-^^^^^^^^^^^^^^^^
-
-This extension will fix nonanticipative variables at their bound if they are at
-their bound in the RelaxedPHSpoke for that subproblem. It will similarily unfix
-nonanticipative variables which are not at their bounds in the RelaxedPHSpoke.
-Because different nonanticipative variables are fixed in different suproblems,
-it will also unfix nonanticipative variables if their value is *not* at the the current
-consensus solution xbar (because the variable was not fixed in a different subproblem
-and therefore came off its bound).
-
 xhat
 ^^^^
 
@@ -194,7 +154,7 @@ CoeffRho
 ^^^^^^^^
 
 Set per variable rho values proportional to the cost coefficient on each non-anticipative variable,
-with an optional multiplier (default = 1.0) that is applied to the computed value. If the coefficient is 0, the default rho value is used instead.
+with an optional multiplier (default = 1.0). If the coefficient is 0, the default rho value is used instead.
 
 primal_dual_rho
 ^^^^^^^^^^^^^^^
@@ -228,7 +188,7 @@ There are options in ``cfg`` to control dynamic updates.
 mult_rho_updater
 ^^^^^^^^^^^^^^^^
 
-This extension does a simple multiplicative update of rho; consequently, the update is cumulative.
+This extension does a simple multiplicative update of rho.
 
 cross-scenario cuts
 ^^^^^^^^^^^^^^^^^^^

doc/src/grad_rho.rst

@@ -1,13 +1,9 @@
 Gradient-based rho
 ==================
-You can find a detailed example using this code in ``examples.farmer.CI.farmer_rho_demo.py``.
 
-Grad-rho is also supported in ``generic_cylinders.py``.
+``mpisppy.utils.gradient.py`` computes rho 
+using the gradient and writes them in a file. 
 
-There are options in ``cfg`` to control dynamic updates:
+.. Note::
+   This only works for two-stage problems for now.
 
-* ``--dynamic-rho-primal-crit`` and ``--dynamic-rho-dual-crit`` are booleans that trigger dynamic rho
-* ``--dynamic-rho-primal-thresh`` and ``--dynamic-rho-dual-thresh`` control how sensitive the trigger is.
-  They have default values, so do not need to be set. See
-  ``dyn_rho_base.py`` to see how the update is done if you don't like the default values.
-* ``--grad-rho-multiplier`` is a cummulative multiplier when rho is set or updated.

doc/src/hubs.rst

@@ -47,13 +47,6 @@ this value (default 1e-1).
 If only the binary terms should be 
 approximated, the option `linearize_binary_proximal_terms` can be used. 
 
-PHPrimal
---------
-This is exactly like the PH hub, except it does not send W values.
-It can be useful when some other cylinder is providing W values
-(e.g., PHDualSpoke, RelaxedPHSpoke).
-
-
 lshaped
 -------
 
@@ -79,15 +72,6 @@ also supplies x and/or W values at every iteration, and is largely based
 on the PH implementation. It utilizes a constant step size rule based on
 `rho` unless modified by an extension.
 
-FWPH
-----
-
-The Frank-Wolfe progressive hedging hub can be used with most spokes
-because it supplies x and/or W values as part of its solution process.
-While FWPH is not known to converge to a primal solution for a SMIP, it
-often discovers excellent incumbent values along the way (when paired with
-an xhat spoke).
-
 Hub Convergers
 --------------