Merge pull request #544 from DLWoodruff/moderner_Bauer

Moderner bauer

Author: bknueven

doc/src/amalgamator.rst

@@ -3,7 +3,11 @@
 Amalgamator
 ===========
 
-For simple problems that do not need extra specification, ``amalgamator.py``
+.. Note::
+   This is an advanced topic, probably of interest only to maintainers of
+   confidence interfal and related code.
+
+The file ``amalgamator.py``
 provides several drivers to solve a problem without writing a program
 that creates the cylinders one by one. ``amalgamator.from_module`` enables
 a high-level user to create a hub-and-spoke architecture using the command 
@@ -49,7 +53,9 @@ It must contains the following attributes for use with cylinders:
 
 
 Create Amalgamator from a module and command line
-------------------------------------------------- Given an options
+-------------------------------------------------
+
+Given an options
 ``Config`` object (typically called `cfg`) as above,
 ``amalgamator.Amalgamator_parser`` creates calls the appropriate
 functions to add the necessary information for different modules.
@@ -88,7 +94,7 @@ Examples
 --------
 
 As intended, the examples of use of Amalgamator are quite short. The first
-example, ``farmer_ama.py``, solves directly the EF. The model can be specified,
+example, ``examples.farmer.archive.farmer_ama.py``, solves directly the EF. The model can be specified,
 e.g. by taking an integer version of it. This specification can be made via
 the command line, thanks to the ``inparser_adder`` method of ``farmer.py``.
 

doc/src/confidence_intervals.rst

@@ -57,7 +57,7 @@ This object has a ``run`` method that returns a gap estimator and a confidence i
 Examples
 --------
 
-There are example scripts for sequential sampling in both ``farmer`` and ``aircond``.
+There are example scripts for sequential sampling in both ``farmer/CI`` and ``aircond``.
 
 Using stand-alone ``mmw_conf.py``
 ---------------------------------
@@ -79,7 +79,7 @@ to be able to pass problem-specific args down without knowing what they are.
 
 Once a model satisfies the requirement for amalgamator, next a ``.npy`` file should be constructed from the given model. This can be accomplished, for example, by adding the line 
 ``sputils.ef_ROOT_nonants_npy_serializer(instance, 'xhat.npy')`` after solving the ef ``instance``. When using ``Amalgamator`` to solve the program, this can be done by adding the line
-``sputils.ef_ROOT_nonants_npy_serializer(ama_object.ef, "xhat.npy")`` to your existing program (see the example in ``farmer.py`` for an example of this).
+``sputils.ef_ROOT_nonants_npy_serializer(ama_object.ef, "xhat.npy")`` to your existing program (see the example in ``examples/farmer/archive/farmer.py`` for an example of this).
 
 Once this is accomplished, on the command line, run
 ``python -m mpisppy.confidence_intervals.mmw_conf my_model.py xhat.npy gurobi --num-scens n --alpha 0.95``. Note that ``xhat.npy`` is assumed to be in the same directory as ``my_model.py`` in this case. If the file is saved elsewhere then the corresponding path should be called on the command line.

doc/src/examples.rst

@@ -47,7 +47,7 @@ constraints are the number of tons per acre that each crop will yield (2.5 for
 wheat, 3 for corn, and 20 for sugar beets).
 
 
-The following code creates an instance of the farmer's model:
+The following code in ``examples/farmer/archive/farmer.py`` (with similar code in ``examples/farmer/farmer.py``) creates an instance of the farmer's model:
 
 .. testcode::
 
@@ -109,7 +109,8 @@ yields for each crop. We can solve the model:
 The optimal objective value is:
 
 .. testoutput::
-
+  :options: +SKIP
+	    
     -118600.0
 
 In practice, the farmer does not know the number of tons that each crop will
@@ -206,136 +207,39 @@ farmer's stochastic program.
 Solving the Extensive Form
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The simplest approach is to solve the extensive form of the model directly.
-MPI-SPPy makes this quite simple:
-
-.. testcode::
+The simplest approach is to solve the extensive form of the model directly. Assuming you are in the directory
+``examples/farmer`` the following unix command will work.
 
-    from mpisppy.opt.ef import ExtensiveForm
+.. code-block:: bash
 
-    options = {"solver": "cplex_direct"}
-    all_scenario_names = ["good", "average", "bad"]
-    ef = ExtensiveForm(options, all_scenario_names, scenario_creator)
-    results = ef.solve_extensive_form()
+    python ../../mpisppy/generic_cylinders.py --module-name farmer --num-scens 3 --EF --EF-solver-name gurobi
 
-    objval = ef.get_objective_value()
-    print(f"{objval:.1f}")
+We can extract the optimal solution itself using the ``--solution-base-name`` option:
 
+.. code-block:: bash
 
-.. testoutput::
-
-    ...
-    -108390.0
+    python ../../mpisppy/generic_cylinders.py --module-name farmer --num-scens 3 --EF --EF-solver-name gurobi --solution-base-name farmersol
 
-We can extract the optimal solution itself using the ``get_root_solution``
-method of the ``ExtensiveForm`` object:
-
-.. testcode::
-
-    soln = ef.get_root_solution()
-    for (var_name, var_val) in soln.items():
-        print(var_name, var_val)
-
-.. testoutput::
-    
-    X[BEETS] 250.0
-    X[CORN] 80.0
-    X[WHEAT] 170.0
+This command writes solution data for nonanticipative variables to two files with the base name farmersol and full scenario solutions to a directory named farmersol_soldir.
 
+.. note::
+   Most command line options relevant to the EF start with --EF. Most other command line options will be silently ignored
+   if ``--EF`` is specified (one exception is ``--solution-base-name``).
+   
 
 Solving Using Progressive Hedging (PH)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-We can also solve the model using the progressive hedging (PH) algorithm.
-First, we must construct a PH object:
-
-.. testcode::
-
-    from mpisppy.opt.ph import PH
-
-    options = {
-        "solver_name": "cplex_persistent",
-        "PHIterLimit": 5,
-        "defaultPHrho": 10,
-        "convthresh": 1e-7,
-        "verbose": False,
-        "display_progress": False,
-        "display_timing": False,
-        "iter0_solver_options": dict(),
-        "iterk_solver_options": dict(),
-    }
-    all_scenario_names = ["good", "average", "bad"]
-    ph = PH(
-        options,
-        all_scenario_names,
-        scenario_creator,
-    )
-
-
-.. testoutput::
-    :hide:
-
-    ...
-
-Note that all of the options in the ``options`` dict must be specified in order
-to construct the PH object. Once the PH object is constructed, we can execute
-the algorithm with a call to the ``ph_main`` method:
-
-.. testcode::
-
-    ph.ph_main()
-
-.. testoutput::
-    :hide:
-
-    ...
-
-
-.. testoutput::
-    :options: +SKIP
-
-
-    [    0.00] Start SPBase.__init__
-    [    0.01] Start PHBase.__init__
-    [    0.01] Creating solvers
-    [    0.01] Entering solve loop in PHBase.Iter0
-    [    2.80] Reached user-specified limit=5 on number of PH iterations
+Here is a simple command that uses PH as the hub algorithm and
+computes lower bounds using a Lagrangian spoke (``--lagrangian``) with
+upper bounds computed by randomly trying scenario solutions to fix the nonanticipative variables (``--xhatshuffle``).
 
-Note that precise timing results may differ.  In this toy example, we only
-execute 5 iterations of the algorithm. Although the algorithm does not converge
-completely, we can see that the first-stage variables already exhibit
-relatively good agreement:
 
-.. testcode::
+.. code-block:: bash
+		
+    mpiexec -np 3 python -m mpi4py ../../mpisppy/generic_cylinders.py --module-name farmer --num-scens 3 --solver-name gurobi_persistent --max-iterations 10 --max-solver-threads 4 --default-rho 1 --lagrangian --xhatshuffle --rel-gap 0.01 
 
-    variables = ph.gather_var_values_to_rank0()
-    for (scenario_name, variable_name) in variables:
-        variable_value = variables[scenario_name, variable_name]
-        print(scenario_name, variable_name, variable_value)
-
-.. testoutput::
-    :hide:
-
-    ...
-    average X[BEETS]
-    ...
-
-.. testoutput::
-    :options: +SKIP
 
-    good X[BEETS] 280.6489711937925
-    good X[CORN] 85.26131687116064
-    good X[WHEAT] 134.0897119350402
-    average X[BEETS] 283.2796296293019
-    average X[CORN] 80.00000000014425
-    average X[WHEAT] 136.72037037055298
-    bad X[BEETS] 280.64897119379475
-    bad X[CORN] 85.26131687116226
-    bad X[WHEAT] 134.08971193504266
-
-The function ``gather_var_values_to_rank0`` can be used in parallel to collect
-the values of all non-anticipative variables at the root. In this (serial)
-example, it simply returns the values of the first-stage variables.
 
 Solving Using Benders' Decomposition
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

doc/src/quick_start.rst

@@ -7,9 +7,9 @@ If you installed from github, install from source using pip in editable mode fro
    
    pip install -e .
 
-This step is not needed if you installed using pip.
-You can also include the extras flags ``mpi`` to install a compliant version
-of mpi4py or ``docs`` to install documentation dependencies from pip.
+This step is not needed if you installed using pip, but we recommend against installing using pip because
+the software is under active development and the pip version is almost always out of date.
+You can also include the extras flag ``docs`` to install documentation dependencies from pip.
 
 
 Verify installation
@@ -25,13 +25,8 @@ terminal commands:
 
    cd examples
    cd farmer
-   python farmer_ef 1 3 solver_name
+   python ../../mpisppy/generic_cylinders.py --module-name farmer --help
 
-but replace `solver_name` with the name of the solver you have installed, e.g., if you have installed glpk, use
-
-::
-   
-   python farmer_ef 1 3 glpk
 
 If you intend to use any parallel features, you should verify that you
 have a *proper* installation of MPI and ``mpi4py``; see the section
@@ -50,15 +45,10 @@ first thing is to code a scenario creation function. See
 If you create a few more helper functions
 (see :ref:`helper_functions`),
 you can make use of the ``generic_cylinders`` program (see :ref:`generic_cylinders`) to use the hub and spoke system or to solve the the EF directly.
-     
-Alternatively, once you have the scenario creation function,
-you can mimic the code in ``examples.farmer.farmer_ef`` to
-solve the extensive form directly. If you want to use the hub
-and spoke system to solve your problem via decomposition, you
-should proceed to the section on writing :ref:`Drivers`, or to
-the :ref:`Examples` section, or to the :ref:`generic_cylinders` section.
-
 
+You might want to start with the farmer example. See the `farmer` directory in the `examples` directory for the
+files `farmer.py` and `farmer_generic.bash`. For more information see :ref:`Examples`.
+     
 PySP Users
 ----------
 
@@ -74,12 +64,6 @@ Here are the general steps:
 
 # Construct a ``PySPModel`` object giving its constructor information about your PySP model.
 
-# Create an options dictionary.
-
-# Create a PH or EF ``mpi-sppy`` object.
-
-# Call its main function.
-
 These steps alone will not result in use of the hub-spoke features of
 `mpi-sppy`, but they will get your PySP model running in
 ``mpi-sppy``. See ``examples/farmer/from_pysp`` for some
@@ -95,4 +79,4 @@ The quickest thing to do is to run one of the canned examples that
 comes with ``mpi-sppy``. They are in subdirectories of
 ``examples`` and sample commands can be obtained by looking at
 the code in ``examples.runall.py``. There is a table in the
-mpi-sppy paper that gives references for some of the examples.
+mpi-sppy paper in MPC that gives references for some of the examples.

doc/src/seqsamp.rst

@@ -27,8 +27,8 @@ For multi-stage, use `multi_seqsampling.py`.
 Examples
 --------
 
-There is sample code for two-stage, sequential sampling in ``examples.farmer.farmer_seqsampling.py`` and
-a bash scrip to test drive it is ``examples.farmer.farmer_sequential.bash``.
+There is sample code for two-stage, sequential sampling in ``examples.farmer.CI.farmer_seqsampling.py`` and
+a bash scrip to test drive it is ``examples.farmer.CI.farmer_sequential.bash``.
 
 There is sample code for multi-stage, sequential sampling in ``examples.aircond.aircond_seqsampling.py`` and
 a bash scrip to test drive it is ``examples.aircond.aircond_sequential.bash``.

doc/src/spokes.rst

@@ -68,9 +68,9 @@ so as to obtain better outer (lower when minimizing) bounds. It can also provide
 a Lagrangian bound even if the hub does not provide lagrangian multipliers.
 
 The easiest way to use ``ph_ob`` is via the vanilla ``ph_ob_spoke`` method
-as illustrated in ``examples.farmer_cylinders.py``. This method takes values
+as illustrated in ``examples.farmer.archive.farmer_cylinders.py``. This method takes values
 from the config object (assuming the config object's ``ph_ob`` method
-was called as shown in the function ``examples.farmer_cylinders._parse_args``)
+was called as shown in the function ``examples.farmer.archive.farmer_cylinders._parse_args``)
 and sets up the options for the spoke.
 
 The option ``ph-ob-initial-rho-rescale-factor`` defaults to 0.1, so if nothing

doc/src/zhat.rst

@@ -13,14 +13,14 @@ Unless you are directly using mid-level functionality, your
 code may need to write the root node nonanticipative variable values
 (called `xhat` or `xhat_one`) to a file for later processing. This is
 typically done using ``sputils.ef_ROOT_nonants_npy_serializer``, which
-is shown in various examples, e.g., ``examples.farmer.farmer.py``
+is shown in various examples, e.g., ``examples.farmer.archive.farmer.py``
 
 zhat4xhat
 ---------
 
 The program ``zhat4xhat`` estimates approximate confidence intervals
 for the objective function value, zhat, given an xhat. See
-``examples.farmer.farmer_zhat.bash`` for a bash script that first
+``examples.farmer.CI.farmer_zhat.bash`` for a bash script that first
 creates the xhat file, then computes an out-of-sample confidence
 interval for it. Note: this program does not compute a confidence
 interval for zstar, which is done using software documented in

examples/afew.py

@@ -41,11 +41,14 @@ def do_one(dirname, progname, np, argstring):
             badguys[dirname] = [runstring]
         else:
             badguys[dirname].append(runstring)
-    os.chdir("..")
+    if '/' not in dirname:
+        os.chdir("..")
+    else:
+        os.chdir("../..")   # hack for one level of subdirectories
 
 
 # for farmer, the first arg is num_scens and is required
-do_one("farmer", "farmer_cylinders.py", 3,
+do_one("farmer/archive", "farmer_cylinders.py", 3,
        "--num-scens=3 --bundles-per-rank=0 --max-iterations=50 "
        "--default-rho=1 --sep-rho --display-convergence-detail "
        "--solver-name={} --xhatshuffle --lagrangian --use-norm-rho-updater".format(solver_name))

examples/farmer/CI/Readme.rst

@@ -0,0 +1,4 @@
+CI
+--
+
+This directory contains code related to getting confidence intervals for the farmer example.