Add minimal-versions CI check

[doxxx93]

Motivation

After #1939, lower bounds declared in Cargo.toml can silently drift from what actually compiles. Since we don't use lockfiles, there's no safety net to catch this.

Solution

Add a minimal-versions job to features.yml that resolves all dependencies to their minimum declared versions and checks that the workspace still compiles.

• cargo hack --remove-dev-deps --workspace to prevent dev-deps from affecting resolution
• cargo +nightly update -Z minimal-versions to lock to lowest allowed versions
• cargo check --workspace --all-features to verify compilation

Also adds a just minimal-versions recipe for local use.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.2%. Comparing base (5c97036) to head (1783184).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #1940     +/-   ##
=======================================
- Coverage   76.3%   76.2%   -0.0%     
=======================================
  Files         89      89             
  Lines       8487    8487             
=======================================
- Hits        6468    6466      -2     
- Misses      2019    2021      +2     

see 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[doxxx93]

@clux I put this as a separate job in features.yml since it needs nightly (unlike the existing cargo-hack job on stable), so merging them into one job wasn't practical.

One thing I'm wondering: should this run on PRs (ci.yml) instead of just on main push? The original issue (#1938) came from a code change PR (#1933) that used a newer API without bumping the minimum version — if minimal-versions had run on that PR, it would have been caught before merge. It would also catch dependabot PRs that bump versions breaking minimal-versions compatibility. It's just a single cargo check, so it shouldn't add much CI time.

What do you think?

[clux]

@clux I put this as a separate job in features.yml since it needs nightly (unlike the existing cargo-hack job on stable), so merging them into one job wasn't practical.

yeah, makes sense.

One thing I'm wondering: should this run on PRs (ci.yml) instead of just on main push? The original issue (#1938) came from a code change PR (#1933) that used a newer API without bumping the minimum version — if minimal-versions had run on that PR, it would have been caught before merge. It would also catch dependabot PRs that bump versions breaking minimal-versions compatibility. It's just a single cargo check, so it shouldn't add much CI time.

What do you think?

mh, yeah, it would actually be good to have this pre-merge. maybe a check / minimal-versions would be a good initial name/grouping for it (check.yml).
EDIT: or maybe it can be put into the lint group initially. i don't know. wdyt?

[doxxx93]

I think ci.yml would be the best fit for now — it already has other check-style jobs like msrv (cargo check on minimum Rust version) and doc (cargo doc build check), which share the same "does it compile under X constraint" nature as minimal-versions.

If we want a dedicated check.yml later, we could move msrv, doc, and minimal-versions together at that point.

What do you think about putting it in ci.yml?

---

Another thought: if we eventually create a check.yml, we could set it up as a prerequisite using needs — so heavier jobs (tests, integration) only run after the check jobs pass. That way we'd fail fast on compilation issues.

[clux]

ci.yml sounds good. needs dependency also makes sense to save some CI time. 👍

[doxxx93]

@clux The minimal-versions check is failing because --all-features enables aws-lc-rs, which pulls in aws-lc-sys v0.21.0 (its minimal version) that doesn't compile with current Rust.

Since this is an upstream issue (hyper-rustls → rustls → aws-lc-rs → aws-lc-sys chain), I'm thinking of using cargo hack check --each-feature --no-private -p kube --skip=aws-lc-rs,oauth,oidc instead of cargo check --all-features — similar to how just hack already skips oauth,oidc.

Does that sound reasonable?

[clux]

oh, convenient, it's actually failing now.

[clux]

Since this is an upstream issue (hyper-rustls → rustls → aws-lc-rs → aws-lc-sys chain), I'm thinking of using cargo hack check --each-feature --skip=aws-lc-rs instead of cargo check --all-features — similar to how just hack already skips oauth,oidc.

hmm, we skip that in the feature powerset job purely because it's impractically large/long runtime otherwise.

but i guess there's nothing that we can actually bump to fix this as it's not a direct dependency for us so skipping it still makes sense. (unless we took an explicit dep on aws-lc-rs and bumped two patch versions, but that's not something we should do without a lockfile)

[doxxx93]

@clux After more local testing on the actual repo, --each-feature keeps hitting upstream minimal-versions failures:

• aws-lc-rs → aws-lc-sys v0.21.0 build failure
• gzip → crc32fast v1.1.0 build failure
• runtime → num-bigint v0.4.0 build failure
• oauth, oidc → need TLS stack (same as just hack)
• k8s-openapi → needs K8S_OPENAPI_ENABLED_VERSION env var since --remove-dev-deps strips the version feature

The skip list keeps growing, which reduces coverage.

I see two options:

1. --each-feature with skips — more coverage but maintains a growing skip list
2. cargo check -p kube (default features only) — simple, still catches issues like tower-http dependency should be updated to 0.6.4 #1938, and can expand later as upstream crates fix their minimum versions

What's your preference?

(Apologies for not testing on the actual repo earlier — I only ran it on a minimal test repo initially.)

---

Update: Verified option 2 passes locally. This also means cargo-hack is only needed for --remove-dev-deps, not for the check step itself.

---

Update 2: Found a better approach — -Z direct-minimal-versions instead of -Z minimal-versions.

The difference: -Z minimal-versions pins all deps (including transitive) to minimum, which is why upstream crates like aws-lc-sys, crc32fast, num-bigint break. -Z direct-minimal-versions only pins direct dependencies to minimum while resolving transitive deps normally.

This means:

• No skip list needed
• --all-features works
• Still catches the exact class of bugs like tower-http dependency should be updated to 0.6.4 #1938

When I tested locally, it actually found 11 lower bounds in Cargo.toml that were inconsistent (e.g. hyper = "1.2.0" but hyper-util 0.1.16 requires hyper >= 1.6.0). These would be fixed alongside.

Updated recipe:

minimal-versions:
  cargo hack --remove-dev-deps --workspace
  cargo +nightly update -Z direct-minimal-versions
  K8S_OPENAPI_ENABLED_VERSION=1.31 cargo check --all-features

Note: K8S_OPENAPI_ENABLED_VERSION is unavoidable here — k8s-openapi version features (e.g. latest) are only enabled via [dev-dependencies], which --remove-dev-deps strips. The env var is the intended fallback in k8s-openapi's build.rs for exactly this case.

[clux]

Ah, yes. Direct minimal sounds like a much better and more maintainable approach for a library without a lockfile! Let's do that, and bump the pins to the new required minimal versions while we are at it!

[clux]

Amazing. Thank you!

[doxxx93]

This turned out to be a much deeper rabbit hole than expected, but happy with where it landed. With -Z direct-minimal-versions, we get accurate lower bound validation without fighting upstream transitive failures.

Going forward, when k8s-openapi bumps its supported Kubernetes version, we just update K8S_OPENAPI_ENABLED_VERSION in the recipe — and the CI will catch any stale lower bounds before they hit users.

[clux]

Going forward, when k8s-openapi bumps its supported Kubernetes version, we just update K8S_OPENAPI_ENABLED_VERSION in the recipe — and the CI will catch any stale lower bounds before they hit users.

ah, right. do you mind putting that in the justfile where we are doing just bump-k8s in that case? (can be an sd call)

[doxxx93]

no problem!

[clux]

great!

[doxxx93]

Definitely the toughest PR I've worked on recently 😅