diff --git a/docs/everest/example.rst b/docs/everest/example.rst index e16be52b09e..9914f28612d 100644 --- a/docs/everest/example.rst +++ b/docs/everest/example.rst @@ -4,20 +4,22 @@ Example ******* -In this section we will go through the setup and running of a simple everest optimization case. +In this section we will go through the setup and running of a simple Everest optimization +case. Problem definition ################## -The optimization problem we will solve using everest will be to minimize the distance between an initial 3D point -and another given target 3D point. +The optimization problem we will solve using Everest will be to minimize the distance +between an initial 3D point and another given target 3D point. Everest config ################## -The main element an Everest optimization experiment requires is a configuration file. Everest supports the `YAML `_ -format for the configuration file. The configuration file is build up of multiple sections and next we will go through it section -by section. +The main element an Everest optimization experiment requires is a configuration file. +Everest supports the `YAML `_ format for the +configuration file. The configuration file is built up of multiple sections and next we +will go through it section by section. Creating the optimization folder:: @@ -77,11 +79,12 @@ everest_config.yml:: min: -1.0 max: 1.0 -``perturbation_magnitude`` controls the size of perturbations (standard deviation of a normal distribution) of controls used to approximate the gradient. -The value depends on the type of control and magnitude of the variables. -For continuous controls smaller values should give a better gradient, -whilst for more discrete controls larger values should give a better -result. However, this is a balance as too large or too small values also cause issues. +``perturbation_magnitude`` controls the size of perturbations (standard deviation of a +normal distribution) of controls used to approximate the gradient. The value depends on +the type of control and magnitude of the variables. For continuous controls smaller values +should give a better gradient, whilst for more discrete controls larger values should give +a better result. However, this is a balance as too large or too small values also cause +issues. ``variables``: list of control variables. @@ -107,17 +110,18 @@ everest_config.yml:: **Details** ``algorithm``: the algorithm used for the optimization -currently supported algorithms: optpp_q_newton, conmin_mfd, conmin_frcg, mesh_adaptive_search. +currently supported algorithms: optpp_q_newton, conmin_mfd, conmin_frcg, +mesh_adaptive_search. ``convergence_tolerance``: provides a real value for controlling the termination of -iteration. In most cases, it is a relative convergence tolerance for the -objective function; i.e., if the change in the objective function between -successive iterations divided by the previous objective function is less than -the amount specified by convergence_tolerance, then this convergence criterion is satisfied on the -current iteration. Since no progress may be made on one iteration followed by significant progress -on a subsequent iteration, some libraries require that the convergence tolerance -be satisfied on two or more consecutive iterations prior to termination of -iteration (from the Dakota Manual.) +iteration. In most cases, it is a relative convergence tolerance for the objective +function; i.e., if the change in the objective function between successive iterations +divided by the previous objective function is less than the amount specified by +convergence_tolerance, then this convergence criterion is satisfied on the current +iteration. Since no progress may be made on one iteration followed by significant progress +on a subsequent iteration, some libraries require that the convergence tolerance be +satisfied on two or more consecutive iterations prior to termination of iteration (from +the Dakota Manual.) Install jobs definition ----------------------- @@ -132,7 +136,8 @@ everest_config.yml:: **Details** -The custom job can be *installed* by adding information regarding it in the **install_jobs** section. Each custom job entry will contain: +The custom job can be *installed* by adding information regarding it in the +**install_jobs** section. Each custom job entry will contain: ``name``: the name the job, the same name will be used in the forward model section. @@ -149,8 +154,8 @@ everest_config.yml:: **Details** -``realizations``: list of realizations to use in optimization ensemble. Typically, this is a -list [0, 1, ..., n-1] of all realizations in the ensemble. +``realizations``: list of realizations to use in optimization ensemble. Typically, this is +a list [0, 1, ..., n-1] of all realizations in the ensemble. Forward model definition ------------------------ @@ -162,11 +167,12 @@ everest_config.yml:: **Details** -``distance3d``: the name of the custom job installed in the ``install_jobs`` section of the config file. -The job name is followed by named arguments the job is designed to accept. +``distance3d``: the name of the custom job installed in the ``install_jobs`` section of +the config file. The job name is followed by named arguments the job is designed to +accept. -``point.json``: file generated by the Everest optimization experiment. It contains the list of control variables with -the values used for the current forward model evaluation. +``point.json``: file generated by the Everest optimization experiment. It contains the +list of control variables with the values used for the current forward model evaluation. Example:: @@ -176,8 +182,8 @@ Example:: "z" : 0 } -Everest creates the file with the name ``point.json`` because we have added to the ``controls`` section the control with -the name ``point``. +Everest creates the file with the name ``point.json`` because we have added to the +``controls`` section the control with the name ``point``. ``distance``: file required for the optimization experiment to succeed. @@ -187,7 +193,8 @@ One of the jobs in the forward model needs to create the file ``distance``. ``distance`` needs to contain the value for the objective function evaluation. -The files needs to be called ``distance`` because we have defined the objective function named ``distance``. +The files needs to be called ``distance`` because we have defined the objective function +named ``distance``. Optimization environment definition ----------------------------------- @@ -252,14 +259,14 @@ everest_config.yml:: output_folder: everest_optimization random_seed: 999 -More information regarding all the available section options and additional sections not covered in the current example -can be found in :ref:`cha_config` section +More information regarding all the available section options and additional sections not +covered in the current example can be found in :ref:`cha_config` section Creating the custom job ####################### -Before we are able to start the optimization experiment we need to create the script for the custom job -``distance3d`` used in the forward model. +Before we are able to start the optimization experiment we need to create the script for +the custom job ``distance3d`` used in the forward model. Creating the folders and files:: @@ -268,8 +275,8 @@ Creating the folders and files:: ~/everest_example/jobs$: touch distance3d.py ~/everest_example/jobs$: chmod a+x distance3d.py -``chmod +x distance3d.py``: command is used to change the access permissions for the file ``distance3d.py``, such that execution of the file is -allowed. +``chmod +x distance3d.py``: command is used to change the access permissions for the file +``distance3d.py``, such that execution of the file is allowed. distance3d.py:: @@ -318,13 +325,14 @@ distance3d.py:: if __name__ == "__main__": main(sys.argv[1:]) -More information about creating custom jobs can be found in the :ref:`cha_creating_custom_jobs` section +More information about creating custom jobs can be found in the +:ref:`cha_creating_custom_jobs` section -Running everest +Running Everest ############### -Now we have all the components needed to start the optimization experiment, -which can be done using the following command:: +Now we have all the components needed to start the optimization experiment, which can be +done using the following command:: ~/everest_example$: everest run everest_config.yml diff --git a/docs/everest/introduction.rst b/docs/everest/introduction.rst index 876ae071057..ef2501856ca 100644 --- a/docs/everest/introduction.rst +++ b/docs/everest/introduction.rst @@ -9,38 +9,75 @@ Background and motivation Everest is a tool, which uses ensembles of models to deliver a robust optimization. -While the result of an data assimilation process is an ensemble of models, the result of the robust optimization process is one optimal strategy, for example for well order or drainage strategy. With this optimised strategy we can, e.g., maximize net present value (NPV) with respect to the geological uncertainty in the models. Note that Everest is a generic tool that can be applied to other optimisation problems (also outside the domain of Oil & Gas). +While the result of a data assimilation process is an ensemble of models, the result of +the robust optimization process is one optimal strategy, for example for well order or +drainage strategy. With this optimized strategy we can, e.g., maximize net present value +(NPV) with respect to the geological uncertainty in the models. Note that Everest is a +generic tool that can be applied to other optimization problems (also outside the domain +of Oil & Gas). Everest ======= -The objectives (what to optimise) can, for example, be the Net Present Value (NPV), the recovery factor, or carbon emissions. The controls (what to adjust in order to optimise the objective) can, for example, be the drilling order of the wells and production and/or injection rates (drainage strategy). Additionally, constraints on these controls can be handled both within Everest and/or in flow simulator. +The objectives (what to optimize) can, for example, be the Net Present Value (NPV), the +recovery factor, or carbon emissions. The controls (what to adjust in order to optimize +the objective) can, for example, be the drilling order of the wells and production and/or +injection rates (drainage strategy). Additionally, constraints on these controls can be +handled both within Everest and/or in flow simulator. -The primary goal of the Everest tool is to find an "optimal" strategies by utilizing an ensemble of reservoir models (e.g., an ensemble of geologically-consistent models). This will help make robust decisions about drilling schedule or drainage strategy in an efficient and assisted manner. +The primary goal of the Everest tool is to find an "optimal" strategy by utilizing an +ensemble of reservoir models (e.g., an ensemble of geologically-consistent models). This +will help make robust decisions about drilling schedule or drainage strategy in an +efficient and assisted manner. -To this end, an optimization problem is formulated and solved by the *Ensemble Optimization* (EnOpt) method described here: :ref:`en-opt-label`. +To this end, an optimization problem is formulated and solved by the +*Ensemble Optimization* (EnOpt) method described here: :ref:`en-opt-label`. -A single configuration file is required to define controls, constraints, objective functions, and other items related to gradient estimation for optimization. This file should also include items related to simulations themselves, such as manipulating schedule files according to the controls and calculating the values of objective functions and constraints. +A single configuration file is required to define controls, constraints, objective +functions, and other items related to gradient estimation for optimization. This file +should also include items related to simulations themselves, such as manipulating schedule +files according to the controls and calculating the values of objective functions and +constraints. -Everest will, then, generate new set of controls as a result of the simulated results, e.g., ensemble objective function values. -The flow chart below provides a detailed description of the elements involved in running an optimization experiment. +Everest will, then, generate new set of controls as a result of the simulated results, +e.g., ensemble objective function values. The flow chart below provides a detailed +description of the elements involved in running an optimization experiment. Considerations regarding the usage of Everest ----------------------------------------------- -Everest can assist the decision makers when there are many outcomes or decisions that need to be made. If the decision and outcome is limited or easy, then Everest may not be needed as one can evaluate the outcomes with a heuristic approach. However, it is unlikely that one can easily find few outcomes or real optimised, robust strategies using this “manual” approach when uncertainty is involved. In the following sections we will discuss the use of Everest in different cases and some of the "ifs and buts " associated with the application of the tool. - -The power of the Everest tool is to assist in optimising strategies while capturing the underlying reservoir uncertainty, which is modelled by an ensemble of models. A crucial factor for the success of the optimisation is to have good enough model quality and the uncertainty well represented - i.e., to span the current understanding or knowledge of the reservoir uncertainty. Moreover, the greater the reservoir complexity, the more important it is to have many model realisations in the ensemble, i.e., to span the actual reservoir uncertainty. The alternative might lead to a situation where the strategy is not representative. The essential take away is that Everest cannot improve your model's quality and your uncertainty representation, and that the quality of the Everest optimisation results is dependent on the quality of the underlying model and uncertainty representation. - -When doing experiments that involve tuning specific parameters, it is advisable to begin with a coarse-grained approach when adjusting control variables. Initially, one might consider modifying the parameters at long intervals, such as every several years or at key points in the process timeline. This sets a foundational structure for the optimization without overwhelming the algorithm with too many variables. - -In general, the following elements should be reflected upon when consider applying Everest for decision support and decision making: - -* the need for, or value of, robustness of optimisation results; - -* the value of time saving from using the tool (heuristic approach versus assisted optimisation); +Everest can assist the decision makers when there are many outcomes or decisions that need +to be made. If the decision and outcome are limited or easy, then Everest may not be +needed as one can evaluate the outcomes with a heuristic approach. However, it is unlikely +that one can easily find few outcomes or real optimized, robust strategies using a +“manual” approach when uncertainty is involved. In the following sections we will discuss +the use of Everest in different cases and some of the "ifs and buts" associated with the +application of the tool. + +The power of the Everest tool is to assist in optimizing strategies while capturing the +underlying reservoir uncertainty, which is modelled by an ensemble of realizations. A +crucial factor for the success of the optimization is to have good enough model quality +and the uncertainty well represented - i.e., to span the current understanding or +knowledge of the reservoir uncertainty. Moreover, the greater the reservoir complexity, +the more important it is to have many model realizations in the ensemble, i.e., to span +the actual reservoir uncertainty. The alternative might lead to a situation where the +strategy is not representative. The essential takeaway is that Everest's results depend +on — but does not improve a model's quality or uncertainty representation. + +When doing experiments that involve tuning specific parameters, it is advisable to begin +with a coarse-grained approach when adjusting control variables. Initially, one might +consider modifying the parameters at long intervals, such as every several years or at key +points in the process timeline. This sets a foundational structure for the optimization +without overwhelming the algorithm with too many variables. + +In general, the following elements should be reflected upon when consider applying Everest +for decision support and decision making: + +* the need for, or value of, robustness of optimization results; + +* the value of time saving from using the tool (heuristic approach versus assisted optimization); * the quality of the underlying reservoir models and uncertainty span; @@ -55,6 +92,19 @@ Everest workflow :width: 700px :alt: Everest workflow -The figure shows the Everest workflow. The workflow starts with defining the initial controls (e.g., drilling order), objective function (e.g., NPV), constraints (e.g., upper injection rate) and related information, such as optimization algorithm, perturbation magnitude, and ensemble realizations used in optimization. Based on these information, Everest will run simulations in order to evaluate either objective function or gradient with respect to the current control variables. Also, it prepares the input files for the simulator, such as the schedule file. - -Normally, different schedule files will be generated for different control values. For example, in case the drilling order is the control, with different drilling orders the well open dates can be totally different, which leads to different schedule files. Once all necessary simulator files are set up, the ensemble of simulations will be run. Based on the simulation results, Everest will calculate the objective function value (e.g. NPV) or constraint values. Everest will then update the controls based on the selected optimization algorithm and run a new experiment until the convergence criteria is satisfied. +The figure shows the Everest workflow. The workflow starts with defining the initial +controls (e.g., drilling order), objective function (e.g., NPV), constraints (e.g., upper +injection rate) and related information, such as optimization algorithm, perturbation +magnitude, and ensemble realizations used in optimization. Based on this information, +Everest will run simulations in order to evaluate either objective function or gradient +with respect to the current control variables. Also, it prepares the input files for the +simulator, such as the schedule file. + +Normally, different schedule files will be generated for different control values. For +example, in case the drilling order is the control, with different drilling orders the +well open dates can be totally different, which leads to different schedule files. Once +all necessary simulator files are set up, the ensemble of simulations will be run. Based +on the simulation results, Everest will calculate the objective function value (e.g. NPV) +or constraint values. Everest will then update the controls based on the selected +optimization algorithm and run a new experiment until the convergence criteria is +satisfied.