Use ./pants bootstrap-cache-key for more reliable setup caching

[huonw]

The setup cache includes symlinks to things outside the cache, such as the (system) Python executable. Thus, the cache should only be reused on machines where those parts of the system exactly match up. This switches from trying to compute an appropriate cache key externally to letting the pants bootstrap script do it itself, via the new PANTS_BOOTSTRAP_TOOLS=1 ./pants bootstrap-cache-key (pantsbuild/setup#128).

This is a breaking change, in that it requires updating the checked in pants scripts to support the new tool.

Fixes #5

[benjyw]

Unfortunately, pyenv will break this, as which will return a path to a shim, which can be different underlying interpreters at different times.

The only reliable thing to do is actually run the python interpreter and get it to report its version.

But a bigger issue here is that the version returned by which python may not be the one the pants script selects, I think. That script has its own interpreter selection logic. And folks can override that logic anyway.

So we may need to have a way for Pants itself to report the interpreter it's running on.

[huonw]

Ah, hm, I think there's potentially two dimensions of issues for a CI cache here:

1. the symlinks to the chosen Python executable (installed somewhere on the system outside Pants' control) in ~/.cache/pants/setup/.../install/bin/
2. the python version matching

The setup-python action for GHA hits both, since the system Python file path ends up containing the exact python version, and thus changes to the GHA runner config can lead to those symlinks breaking.

So we may need to have a way for Pants itself to report the interpreter it's running on.

Maybe I misunderstand what you're imagining, but I think this will be too late for the setup cache? As in, CI will need to be able to extract this info before bootstrapping pants, or else it'll need to bootstrap before it can determine the cache to use to save on the bootstrapping time...

---

Some possible options here (just me brainstorming):

1. make sure this init-pants action to works with the current behaviour of the setup-python action, without necessarily handling cases that might come up in other environments (e.g. pyenv/local dev), meaning the which-based approach of this PR as written may be okay
2. adjust to something that'll handle more cases of python being a shim
   Suggested change

   	PYTHON_VERSION=$(which python)
   	PYTHON_VERSION=$(python -c 'import sys; print(sys.executable)')
3. something even fancier, like having the pants shell script able to report this sort of info before bootstrapping (e.g. ./pants --bootstrap-cache-key or something, that'd print some 'opaque' string like pants_version_2.13.0-python_path_/usr/... and whatever other info is required), which could package up the TOML file parsing too. I don't know how easy it is to propagate changes to that script though.

Since this init-pants action will only ever run in GitHub actions, 1 seems fine to me, but I'm more than happy to take guidance about other approaches.

[benjyw]

The problem is that whatever python is on the path here may not be the one the pants script selects.

So yeah, we may have to have a ./pants cmd that returns the full path to intepreter it selects. This would actually be useful in general, when debugging pants bootstrapping issues.

[benjyw]

In other words, your 3. seems like the way to go.

[huonw]

Cool! I've proposed a possible step towards 3 in pantsbuild/setup#128. I'm imagining this action is then updated to do something like:

    - name: Get Pants bootstrap cache key
      id: pants_version
      shell: bash
      run: |
        if ! grep 'PANTS_BOOTSTRAP_TOOLS' ./pants; then
            echo "This action requires a newer ./pants script. <update instructions here...>"
            exit 1
        fi
        
        PANTS_BOOTSTRAP_CACHE_KEY=$(PANTS_BOOTSTRAP_TOOLS=1 ./pants bootstrap-cache-key)
        echo "::set-output name=pants_bootstrap_cache_key::$PANTS_BOOTSTRAP_CACHE_KEY"

    - name: Cache Pants Setup
      id: cache-pants-setup
      uses: actions/cache@v2
      with:
        path: |
          ~/.cache/pants/setup
        key: pants-setup-${{ steps.pants_version.outputs.pants_bootstrap_cache_key }}

[huonw]

I've made that update, and validated on our repo (seems to work, including the too-old pants script detection).

I've also added a drive-by fix to not run the base-commit look-up when it isn't going to be used, since it was breaking in our repo (due to a lack of gh script, I think).