chore: update hermetic library generation workflow #10693
There are no files selected for viewing
|
Contributor
There was a problem hiding this comment. nit (maybe not in scope): What's the purpose of this script? Maybe a one-liner comment can help.
Contributor
Author
There was a problem hiding this comment. There's a one-liner comment in each job in the workflow.
Contributor
There was a problem hiding this comment. Can you remind me what this file is used for?
Contributor
Author
There was a problem hiding this comment. This workflow is used to verify generated files, e.g.,
Contributor
There was a problem hiding this comment. What if we do want to update
Contributor
Author
There was a problem hiding this comment.
The check will fail. This is a not required check only to remain the author the change is indeed intentional.
root pom and gapic-libraries-bom are generated through hermetic build cli script (just like repo-level post processing). Other checks are done through shell script in this repo, which are not generating any java file, so they are relatively fast. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,27 +1,25 @@ | ||
| name: Hermetic library generation upon generation config change through pull requests | ||
| on: | ||
| pull_request: | ||
| paths: | ||
| - "generation_config.yaml" | ||
|
Contributor
There was a problem hiding this comment. Is the trigger only for main branch? If yes, can this be configured to be triggered for other branches, like a future LTS branch?
Contributor
Author
There was a problem hiding this comment. This can be triggered when a pull request changes Therefore, it can be used in any branch since the pull request is not necessarily based on
Contributor
There was a problem hiding this comment. I don't see any configuration for a target branch, does it mean if I create a PR with |
||
|
|
||
| jobs: | ||
| library_generation: | ||
| runs-on: ubuntu-latest | ||
| env: | ||
| library_generation_image_tag: 2.39.0 | ||
|
Contributor
There was a problem hiding this comment. Is this configured to be updated renovate bot?
Contributor
Author
There was a problem hiding this comment. I plan to update renovate config in separate PR. |
||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
| - name: Generate changed libraries | ||
| shell: bash | ||
| run: | | ||
| set -x | ||
| bash generation/hermetic_library_generation.sh \ | ||
| --target_branch ${{ github.base_ref }} \ | ||
| --current_branch ${{ github.head_ref }} \ | ||
| --image_tag "${library_generation_image_tag}" | ||
| env: | ||
| GH_TOKEN: ${{ secrets.CLOUD_JAVA_BOT_TOKEN }} | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| name: Update googleapis commit | ||
| on: | ||
| schedule: | ||
| - cron: '* 2 * * *' | ||
| workflow_dispatch: | ||
|
|
||
|
|
||
| jobs: | ||
| update-googleapis-commit: | ||
| runs-on: ubuntu-22.04 | ||
| env: | ||
| # the branch into which the pull request is merged | ||
| base_branch: lts-11 | ||
|
||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Update googleapis commit to latest | ||
| shell: bash | ||
| run: | | ||
| set -x | ||
| bash generation/update_googleapis_commit.sh \ | ||
| --base_branch "${base_branch}"\ | ||
| --repo ${{ github.repository }} | ||
| env: | ||
| GH_TOKEN: ${{ secrets.CLOUD_JAVA_BOT_TOKEN }} | ||
|
Contributor
There was a problem hiding this comment. Really like this script! This script is generic enough to be reused by any repo. We need to come up with a way to reused this script, but worst case we can just copy this script to every handwritten repos. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| #!/bin/bash | ||
| # This script should be run at the root of the repository. | ||
| # This script is used to, when a pull request changes the generation | ||
| # configuration (generation_config.yaml by default): | ||
| # 1. compare generation configurations in the current branch (with which the | ||
| # pull request associated) and target branch (into which the pull request is | ||
| # merged); | ||
| # 2. generate changed libraries using library_generation image; | ||
| # 3. commit the change to the pull request | ||
| # 4. edit the pr body with generated pull request description, if applicable. | ||
|
|
||
| # The following commands need to be installed before running the script: | ||
| # 1. git | ||
| # 2. gh | ||
| # 3. docker | ||
|
|
||
| # The parameters of this script is: | ||
| # 1. target_branch, the branch into which the pull request is merged. | ||
| # 2. current_branch, the branch with which the pull request is associated. | ||
| # 3. image_tag, the tag of gcr.io/cloud-devrel-public-resources/java-library-generation. | ||
| # 3. [optional] generation_config, the path to the generation configuration, | ||
| # the default value is generation_config.yaml in the repository root. | ||
| while [[ $# -gt 0 ]]; do | ||
| key="$1" | ||
| case "${key}" in | ||
| --target_branch) | ||
| target_branch="$2" | ||
| shift | ||
| ;; | ||
| --current_branch) | ||
| current_branch="$2" | ||
| shift | ||
| ;; | ||
| --image_tag) | ||
| image_tag="$2" | ||
| shift | ||
| ;; | ||
| --generation_config) | ||
| generation_config="$2" | ||
| shift | ||
| ;; | ||
| *) | ||
| echo "Invalid option: [$1]" | ||
| exit 1 | ||
| ;; | ||
| esac | ||
| shift | ||
| done | ||
|
|
||
| if [ -z "${target_branch}" ]; then | ||
| echo "missing required argument --target_branch" | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [ -z "${current_branch}" ]; then | ||
| echo "missing required argument --current_branch" | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [ -z "${image_tag}" ]; then | ||
| echo "missing required argument --image_tag" | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [ -z "${generation_config}" ]; then | ||
| generation_config=generation_config.yaml | ||
| echo "Use default generation config: ${generation_config}" | ||
| fi | ||
|
|
||
| volume_name="repo" | ||
| workspace_name="/workspace/repo" | ||
| repo_volumes="${volume_name}:${workspace_name}" | ||
| baseline_generation_config="baseline_generation_config.yaml" | ||
| message="chore: generate libraries at $(date)" | ||
|
|
||
| git checkout "${target_branch}" | ||
| git checkout "${current_branch}" | ||
|
Contributor
There was a problem hiding this comment. qq: Has this double
Contributor
Author
There was a problem hiding this comment. In the action, the fetch-depth is 0 so it needs We want to copy |
||
| # copy generation configuration from target branch to current branch. | ||
| git show "${target_branch}":"${generation_config}" > "${baseline_generation_config}" | ||
| diff "${generation_config}" "${baseline_generation_config}" || echo "config diff" | ||
| # bind docker volume to include google-cloud-java in docker running environment. | ||
|
||
| if [[ $(docker volume inspect ${volume_name}) != '[]' ]]; then | ||
| docker volume rm ${volume_name} | ||
| fi | ||
| docker volume create \ | ||
| --name ${volume_name} \ | ||
| --opt "type=none" \ | ||
| --opt "device=$(pwd)" \ | ||
| --opt "o=bind" | ||
| # run hermetic code generation docker image. | ||
| docker run \ | ||
| --rm \ | ||
| -v "${repo_volumes}" \ | ||
| -v /tmp:/tmp \ | ||
| -v /var/run/docker.sock:/var/run/docker.sock \ | ||
| -e "RUNNING_IN_DOCKER=true" \ | ||
| -e "REPO_BINDING_VOLUMES=-v ${repo_volumes}" \ | ||
| gcr.io/cloud-devrel-public-resources/java-library-generation:"${image_tag}" \ | ||
| python /src/cli/entry_point.py generate \ | ||
| --baseline-generation-config-path="${workspace_name}/${baseline_generation_config}" \ | ||
| --current-generation-config-path="${workspace_name}/${generation_config}" \ | ||
| --repository-path="${workspace_name}" | ||
| # commit the change to the pull request. | ||
| [ -z "$(git config user.email)" ] && git config --global user.email "[email protected]" | ||
blakeli0 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| [ -z "$(git config user.name)" ] && git config --global user.name "cloud-java-bot" | ||
| git add java-* pom.xml gapic-libraries-bom/pom.xml versions.txt "${generation_config}" | ||
| git commit --allow-empty -m "${message}" | ||
| git push | ||
| # set pr body if pr_description.txt is generated. | ||
| if [[ -f "pr_description.txt" ]]; then | ||
| pr_num=$(gh pr list -s open -H "${current_branch}" -q . --json number | jq ".[] | .number") | ||
| gh pr edit "${pr_num}" --body "$(cat pr_description.txt)" | ||
| fi | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit:
ommitedshould beomittedThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done.