Skip to content

2. Setup for Local API Development

Jump to bottom
adfarth edited this page Aug 9, 2023 · 37 revisions

The setup steps are this page are meant to be performed sequentially.

Set up your keys

After cloning the API, you must set up your keys.py. Please pull the latest code if you did not just clone the repository. Within the directory into which you cloned the REopt API is a file called keys.py.template. First, make a copy of the keys.py.template file and rename it keys.py (in the same directory). Then, get a NREL Developer API key and replace the DEMO-KEY value within keys.py with your API key.

Setup to evaluate GHP locally (v1-v2 only)

The evaluation of the GHP technology requires a ground heat exchanger (GHX) model which is governed by a more restrictive license than the BSD-3 license which governs REopt_API usage and development. The separation of license terms between REopt_API and the GHX model is accomplished by a git version control feature called git submodule.

If you intend to run GHP in your REopt evaluations, after you clone the REopt_API repository, you need to run these two commands in a terminal inside of the REopt_API directory in order to evaluate the GHP technology:

  1. git submodule init
  2. git submodule update

These two steps effectively clone the GHX model repository into the expected location in the REopt_API to enable it to evaluate GHP while preserving the governance of the separate license terms of using the GHX model.

You do not need to do these steps if you are using v3 of the API.

Get Xpress or other solver

v3 of the REopt API can only currently be used with the Xpress solver. However, if you wish to use v3 capabilities with an alternative solver, you can directly use the REopt.jl (Julia package) with any solver. See REopt.jl documentation for instructions. TODO: edit URL? If you wish to use v1 or v2 of the REopt API, you can use any compatible solver. For users without a commercial solver license, Cbc is a free option (however, Cbc is relatively slow at solving mixed integer problems compared to commercial solvers).

For internal NREL developers: To get Xpress, follow these instructions on NREL's private GitHub server.

For external developers: To use the Xpress solver with the Docker container API setup, you must generate a license file (xpauth.xpr) for the xpress_license_server container using a unique MAC address. This license must be a "Floating" type which is used for license servers, as it is used in this API.

In the docker-compose.yml file located in the top directory, replace the mac_address parameter of the licenseserver service with a randomly generated MAC address in the same format (use this MAC address generator with colon separator and lowercase letters).

Now perform step 2 of the Docker setup for Xpress (docker compose build). After that completes, run (replace Your MAC Address with the one you generated):

docker run --entrypoint /bin/sh --name reopt_lite_api_licenseserver --mac-address Your MAC Address -itd reopt_lite_api_licenseserver

Open a shell in the container by:

docker exec -it reopt_lite_api_licenseserver bash

Run the xphostid tool by:

xphostid

Copy the entire output under the xphostid command and save it into a text file and use that for generating the xpauth.xpr license (floating type) file.

Once you have generated the xpauth.xpr license file, put the file in the julia_src/xpress/licenseserver directory (overwrite the existing one, if there is one)

Install and build docker

Once you have your keys.py and solver set up, install docker (https://www.docker.com/get-started) and create an account so that you are able to pull images from docker hub.

Build docker images for v3 and higher

Before building docker images, make sure the docker application is open.

If using the API with the officially registered REopt Julia package

The officially-registered REopt Julia package is the master branch here and the package imported by defaulted when a user does add REopt in Julia. See the next section if you are instead wanting to access a locally "dev'ed" branch of REopt.jl or a feature branch of REopt.jl on GitHub.

Build docker images, by entering the following in a terminal in the top directory of the REopt_API:

docker compose build

This step is only required the first time or if the images have changes or were removed. This will take some time to build (~10 minutes), and when it finishes you should see the images listed when you type docker images into a terminal or view the images in the Docker Desktop app.

Start the containers (anytime the containers are stopped or removed, or your computer sleeps/hibernates/restarts) with:

docker compose up

This should be pretty quick, but you have to wait for the containers to start running (Check Docker Desktop, another Docker UI, or type docker ps into another terminal to see a list of five running containers).

Once built, all containers should appear as green and "running" in the docker GUI.

If using the API with a REopt.jl feature branch or locally-hosted repository

Spin up docker with a feature branch of the REopt Julia

  1. Make your changes to the REopt Julia package as needed and push your feature branch to GitHub (should appear here; you will need to have write access to do this.)
  2. Point the julia_src environment of the REopt API to the feature branch of the Julia package by running the following from the julia_src directory:
julia --project=.
using Pkg
pkg> add REopt#branchname

(replace branchname with feature branch name from https://github.com/NREL/REopt.jl/branches)

The step above will modify the REopt_API julia_src/Manifest.toml file. Any local API calls or unit tests you run will now use specified feature branch of REopt.jl.

  1. Spin up docker containers by changing back to the top-level directory of REopt_API then do (NREL users may need to suspend VPN):
docker compose build
  1. Start the containers (anytime the containers are stopped or removed, or your computer sleeps/hibernates/restarts) with:
docker compose up

Once built, all containers should appear as green and "running" in the docker GUI.

NOTE: Step 2 will modify the julia_src/Manifest.toml file for the REoptLite entry. DO NOT COMMIT THESE CHANGES (with the exception of instances in which you wish to use GitHub's unit tests with your changes, in which case you should eventually undo them as shown below).

  1. Once you are done testing your local changes to the Julia package, reset the julia_src environment with: pkg> add REopt (from the julia_src directory with that environment activated)

Spin up docker with a local copy of the Julia package

If you do not have write access to REopt.jl or want to use a locally-modified version of REopt.jl, follow these instructions for spinning up docker.

Instead of using the default docker-compose.yml file when developing the Julia package locally in concert with the API, one must use the docker-compose.nojul.yml file for starting the the API components (Redis, Postgresql, Django, Celery, and a solver license server if you have one). As the name implies, the docker-compose.nojul.yml does not start a Julia service. Instead, to test the API with local changes to the Julia package one must start the Julia HTTP service locally in an environment that points towards your local copy of the Julia package.

  1. Make your changes to the Julia package and/or the API code as needed.
  2. docker-compose -f docker-compose.nojul.yml up (This step can really be run at any time, unless you are modifying Celery tasks such as run_jump_model, which requires restarting the Django and Celery services to pick up on changes.)
  3. Point the julia_src environment to your local copy of the Julia package by running the following from the julia_src directory:
julia --project=.
using Pkg
Pkg.develop("REopt")
  1. Exit the Julia session from the previous step and run julia --project=. http.jl (or, in another julia terminal include(http.jl). This will start the Julia HTTP server that will provide the reopt endpoint, which will use your local copy of the REopt Julia package to run_reopt.

NOTE: Step 3 will modify the julia_src/Manifest.toml file for the REoptLite entry. DO NOT COMMIT THESE CHANGES (with the exception of instances in which you wish to use GitHub's unit tests with your changes, in which case you should eventually undo them as shown below).

Once you are done testing your local changes to the Julia package, reset the julia_src environment with:

julia --project=julia_src
using Pkg
Pkg.free("REopt")

Building docker images for v1-v2

You can alternatively use a different solver with REopt API v1-v2. You must specify this solver when building docker images. The examples below are for using Cbc.

To use Cbc, from the top directory within REopt_API:

docker compose -f docker-compose.cbc.yml build

This will take some time to build (~10 minutes), and when it finishes you should see four images listed when you type docker images into a terminal or view the images in the Docker Desktop app.

Start the containers (anytime the containers are stopped or removed, or your computer sleeps/hibernates/restarts) specifying cbc:

docker compose -f docker-compose.cbc.yml up

This should be pretty quick, but you have to wait for the containers to start running (Check Docker Desktop, another Docker UI, or type docker ps into another terminal to see a list of five running containers).

Interactive Commands on Container

To check if all the containers are running as expected, run the following command:

docker ps

You can interactively run commands in a docker container with:

docker exec -it <container-name> /bin/bash

where container names can be found via docker ps. For more see the Docker documentation.

For instructions on running tests when logged into the celery container see Develop and Test the API (v3). TODO: Update

Stop/Remove the docker containers

Docker containers are meant to be ephemeral, only persisting for the required development and testing. When you are finished with them, it is good practice to stop and remove them.

Stop your docker containers with (cntrl/cmd+c).

The command below removes containers, and removes the database which is stored in a docker volume (-v). For Xpress users:

docker compose down -v

For Cbc users:

docker compose -f docker-compose.cbc.yml down -v

To remove all unused containers, networks, images (both dangling and unreferenced), and optionally, volumes, run (see documentation:

docker system prune

The next time you want to run the app, start at the build step if the images have changes or were removed. If not, start at the up step.

Docker setup FAQ and troubleshooting

Out-of-memory issues

If you installed docker-compose via Docker Desktop the default memory limits for the celery container maybe too low (2GB for Mac for example). The optimization process can take more than 3 GB of memory, which will lead to a container killing the process. Therefore, we recommend increasing the Docker memory limit to at least 4 GB. You can do this in the docker GUI via Resources > Advanced.

Address already in use

If you are already running PostgreSQL or Redis when you start the docker containers then you may get a conflict with the ports that PostgreSQL and Redis use. To fix this you can either shut down your PostgreSQL and/or Redis servers or change the port numbers in the docker-compose.yml file. For more on ports see the Docker documentation.

If you are having trouble getting the API setup with Docker after you've attempted it the first time, it is suggested to start from scratch with the images and containers:

Other useful docker commands

Stop and remove all docker containers and volumes For Cbc solver:

docker compose -f docker-compose.cbc.yml down -v

For Xpress solver:

docker compose down -v

Delete all docker images and build cache (WARNING: this will remove any images even if not associated with the REopt API)

docker system prune -a

Remove selected images to avoid wiping non-REopt API images

First get a list of the images

docker images

Then do this for each of the REopt API images:

docker image rm [image_name]

Check that everything is wiped

docker system df

You should see zeros across the board (unless you only removed selected images). There may be a dangling volume still, in which case you can:

docker volume prune

Windows docker image build issue Specifically for Windows operating machines, if you encounter an error which includes “bash^M”, run the following line of code in git bash terminal inside of your repository directory. You may need to repeat for each branch you work on... 

sed -i -e 's/\r$//' bin/wait-for-it.bash 

Clone this wiki locally