Add release process instructions

Author: epwalsh

README.md

@@ -7,3 +7,21 @@ Code for in-loop evaluation tasks used by the OLMo training team.
 ```
 pip install ai2-olmo-eval
 ```
+
+## Release process
+
+### Steps
+
+1. Update the version in `src/olmo_eval/version.py`.
+2. Run the release script:
+
+    ```bash
+    ./src/scripts/release.sh
+    ```
+
+    This will commit the changes to the CHANGELOG and `version.py` files and then create a new tag in git
+    which will trigger a workflow on GitHub Actions that handles the rest.
+
+### Fixing a failed release
+
+If for some reason the GitHub Actions release workflow failed with an error that needs to be fixed, you'll have to delete the tag on GitHub. Once you've pushed a fix you can simply repeat the steps above.

src/scripts/release.sh

@@ -4,7 +4,9 @@ set -e
 
 TAG=$(python -c 'from olmo_eval.version import VERSION; print("v" + VERSION)')
 
-git pull
+git pull > /dev/null
+git tag -l | xargs git tag -d > /dev/null
+git fetch -t > /dev/null
 
 # Make sure tag/release doesn't already exist.
 STATUS_CODE=$(curl -s -o /dev/null -w "%{http_code}" "https://github.com/allenai/OLMo-in-loop-evals/releases/tag/${TAG}")