Support credential helper for pull instead of docker config.json

[peakschris]

I'd like to use bazel credential helper for all image pulls, and not to have to create a docker config file.

The docs suggest that rules_img requires a docker config file, and that credential helper is only available for "some push strategies".

Is that correct?

https://github.com/bazel-contrib/rules_img#credential-discovery

Thanks! Chris

[peakschris]

I've tried both the following, but they do not seem to take effect for pull:

common --@rules_img//img/settings:docker_config_path=out/bazel/container/config.json
common:windows --@rules_img//img/settings:credential_helper=c:/apps/bazel/credential-helper.exe

Repository rule pull defined at:
  D:/ws/b/7ns5ou3z/external/rules_img+/img/private/repository_rules/pull.bzl:233:23: in <toplevel>
ERROR: D:/ws/b/7ns5ou3z/external/rules_img+/img/private/repository_rules/download.bzl:441:13: An error occurred during the fetch of repository '+_repo_rules+vault-windows':
   Traceback (most recent call last):
        File "D:/ws/b/7ns5ou3z/external/rules_img+/img/private/repository_rules/pull.bzl", line 48, column 28, in _pull_impl
                _download_with_tool(
        File "D:/ws/b/7ns5ou3z/external/rules_img+/img/private/repository_rules/download.bzl", line 441, column 13, in download_with_tool
                fail("img tool failed with exit code {} and message {}".format(result.return_code, result.stderr))
Error in fail: img tool failed with exit code 1 and message Failed to download from harbor.outcompany.com: downloading manifest: getting manifest: error getting credentials - err: exec: "docker-credential-desktop": executable file not found in %PATH%, out: ``
Error: Failed to download blob from all registries: downloading manifest: getting manifest: error getting credentials - err: exec: "docker-credential-desktop": executable file not found in %PATH%, out: ``
ERROR: no such package '@@+_repo_rules+ourociimage//': img tool failed with exit code 1 and message Failed to download from harbor.ourcompany.com: downloading manifest: getting manifest: error getting credentials - err: exec: "docker-credential-desktop": executable file not found in %PATH%, out: ``

Putting the credential in %USERPROFILE%.docker\config.json is the only way I can get the pull to work.

Would it be possible to support credential helper and/or local config files?

[malt3]

I think this will require patches in go-containerregistry. It looks like it fails hard if it can't initialize the docker config source.
This also may require some setup to inject the credential helper to the action that downloads layers.
Marking this as a feature request.

[peakschris]

Thank you!

I've tried with 0.3.8, as I found some work in this area had gone into the last release. It's failing with a different error now:

>call bazel build //src/... --output_groups=oci_layout 
INFO: Writing tracer profile to 'D:/ws/b/7ns5ou3z/command.profile.gz'
ERROR: D:/ws/b/7ns5ou3z/external/+_repo_rules+myimage-windows/BUILD.bazel:12:15: DownloadBlob external/+_repo_rules+myimage-windows/sha256_de8e8a3c73e339d2f0dd61c2f6ced005c85ce0ad4b5f552eb3897198f6bf03a8 failed: (Exit 1): img.exe failed: error executing DownloadBlob command (from target @@+_repo_rules+myimage-windows//:layers) external\rules_img++prebuilt_img_tool+img_toolchain_windows_amd64\img.exe download-blob --digest sha256:de8e8a3c73e339d2f0dd61c2f6ced005c85ce0ad4b5f552eb3897198f6bf03a8 --output ... (remaining 2 arguments skipped)
Failed to download from harbor.mycompany.com/project/myimage: getting compressed layer: GET https://harbor.mycompany.com/v2/project/myimage/blobs/sha256:de8e8a3c73e339d2f0dd61c2f6ced005c85ce0ad4b5f552eb3897198f6bf03a8: UNAUTHORIZED: unauthorized to access repository: project/myimage, action: pull: unauthorized to access repository: project/myimage, action: pull
Error: Failed to download blob from all sources: getting compressed layer: GET https://harbor.mycompany.com/v2/project/myimage/blobs/sha256:de8e8a3c73e339d2f0dd61c2f6ced005c85ce0ad4b5f552eb3897198f6bf03a8: UNAUTHORIZED: unauthorized to access repository: project/myimage, action: pull: unauthorized to access repository: project/myimage, action: pull

In this testing, I also found that something is quite aggressively caching the image layers and/or access token. bazel clean --expunge and delete of bazel's cache directory don't seem to impact this. Is the location of this image cache documented anywhere?

[malt3]

There is no secret image cache. This is either writing to the repository cache or to Bazel's output base.

[Harshal96]

@malt3 I can work on this; looks like this needs two pieces:

1. Propagation of --@rules_img//img/settings:credential_helper into pull-time downloads, especially lazy DownloadBlob actions, via IMG_CREDENTIAL_HELPER?
2. Making pull auth prefer/use that helper even when Docker config discovery is broken, like when ~/.docker/config.json references docker-credential-desktop an it is not in %PATH%.

credential_helper does not look consistently wired through the repository rule / module extension / generated download_blobs action path like docker_config_path / REGISTRY_AUTH_FILE is (partially?) wired for lazy blob downloads

Probably some auth plumbing plus a regression test with a fake credential helper and no usable Docker config?

[malt3]

@Harshal96 thanks for offering to work on this! That would be great.
Just some complications to be aware of: any place where we run the pull tool in a repo rule or module extension via rctx.execute or mctx.execute is an obvious place to inject the credential helper path and docker config path via env vars.
I'd do that part first. The second part is a lot harder: lazy DownloadBlob actions may run on your local machine or in RBE. They are build actions. As such, it's not entirely correct to just give them a path to a credential helper that is installed on the machine that Bazel runs on. This path may not exist wherever the action is scheduled. Furthermore, the action may run on a different CPU architecture or OS from your Bazel local host.
In order to truly make DownloadBlob work with a Bazel credential helper, you would need to create a "bazel credential helper toolchain" that has at least some exec constraints.
Then, users need to register that toolchain for their credential helper of choice.
This may be worth it, just be aware that this is a different domain (repo rules and "bazel run" commands all run on the host, build actions may run anywhere).
Something else to be aware of: you probably want to avoid tracking any files in Bazel that (potentially) contain secrets. If we were to track the docker config as a file that Bazel may upload to the remote cache, this needs to be very explicit and opt-in. Tracking the path as an env variable should be fine (although it may be meaningless when the action runs on another machine).

[Harshal96]

@malt3 #524 -- thoughts?