intermittent cache? failures running ko apply

[bobcallaway]

Error: error processing import paths in "config/trillian/updatetree/200-updatetree-job.yaml": error resolving image references: error getting matching image from index: Image("sha256:22cda3953e236576e5d113fc6a9290b27e179fbe5937efb1fdc16d81ddfb6120"): open ~/ko/img/blobs/sha256/2a0679608f5a6374fbe886e22ef1dc4b5578165a97db9132a128c0aa92a72155: no such file or directory

Here's an example of where we've gotten a failure when using ko apply to a kind cluster within a GHA run. It is flaky, as I can re-run it and it may or may not succeed.

https://github.com/sigstore/scaffolding/actions/runs/6764519896/job/18404319467?pr=819

perhaps worth noting: we are setting KOCACHE in for this action, as well as attempting a restore via actions/cache

[jonjohnsonjr]

This should be falling back to the registry, so I've messed something up here. If you add a -v flag to get verbose logs, it might help point out exactly what's wrong, but I'll see if I can figure it out either way.

[jonjohnsonjr]

Wild guess: #1175

[bobcallaway]

https://github.com/sigstore/scaffolding/actions/runs/6774230596/job/18410845671?pr=812 here's a failure with -v added

[jonjohnsonjr]

Apologies for the laziness here, but can you add:

  with:
    version: tip

To your setup-ko to see if that was the issue? If not I'll try to get a local repro and actually debug it.

[bobcallaway]

https://github.com/sigstore/scaffolding/actions/runs/6775027890/job/18414014648 still failing, though a slightly different error message

[bobcallaway]

it did run clean on the first attempt though

[bobcallaway]

friendly ping - any updates on this? It's failing pretty regularly on runs: https://github.com/sigstore/scaffolding/actions/runs/6817870310

[imjasonh]

You seem to still be installing the latest release of ko, and not building from head to get Jon's fix: https://github.com/sigstore/scaffolding/actions/runs/6817870310/workflow#L76

    - uses: ko-build/setup-ko@ace48d793556083a76f1e3e6068850c1f4a369aa # v0.6
      with:
        version: tip

[bobcallaway]

You seem to still be installing the latest release of ko, and not building from head to get Jon's fix: https://github.com/sigstore/scaffolding/actions/runs/6817870310/workflow#L76

    - uses: ko-build/setup-ko@ace48d793556083a76f1e3e6068850c1f4a369aa # v0.6
      with:
        version: tip

you're right, the link i gave this morning was not running from tip - however https://github.com/sigstore/scaffolding/actions/runs/6775027890/job/18414014648 is and it still fails :/

[bobcallaway]

after upgrading to tip, i'm not seeing this anymore, so i'm cool if you want to close this as fixed. do you know when you'll cut the next release?

thanks @jonjohnsonjr for the patch