Added non-conflicting hash for install files

[MarconZet]

Sumary

This commit introduces lock file version 3 with per-artifact hashing instead of a single global hash.

This per-artifact hashing approach can reduce the amount of merge conflicts when multiple people update canonical version in large monorepo.

The code still supports reading v2 lock files - it checks for v3 first, then falls back to v2, then v1. Users with older lock files will see a message to repin.

Key Changes

1. Lock File Format Change (v2 → v3)

• Before (v2): __INPUT_ARTIFACTS_HASH and __RESOLVED_ARTIFACTS_HASH were single integer values
• After (v3): Both are now dictionaries mapping each artifact coordinate to its individual hash

Example in maven_install.json:

// Old format 
"__INPUT_ARTIFACTS_HASH": 1994476565, 
"__RESOLVED_ARTIFACTS_HASH": -274973469,
// New format
"__INPUT_ARTIFACTS_HASH": { "com.google.guava:guava": 733518530, "junit:junit": -652553691, "..." }, 
"__RESOLVED_ARTIFACTS_HASH": { "com.google.guava:guava": -1587873388, "..." }

2. Hash Computation Changes (private/rules/v3_lock_file.bzl:53-108)

The new _compute_lock_file_hash_v3 function computes individual hashes per artifact that include:

• The artifact's own info (coordinates, SHA sums)
• The repository it came from
• Hashes of all transitive dependencies (dependency-aware hashing)

3. Input Hash Changes (private/rules/coursier.bzl:334-386)

compute_dependency_inputs_signature now returns a dictionary of per-artifact hashes plus backward-compatible v1/v2 signatures.

[shs96c]

This is looking really good. I like the idea of only having conflicts if the transitive deps have changed.

[vinnybod]

@shs96c question for a potential breaking change in the next major update.

It seems like this code and the code in v3_lock_file.bzl are similar. IIRC, the reason the starlark implementation exists is if the user doesn't have a lockfile.
If that is the case, is there a possibility to consolidate around the java code (which is easiest to test tbh) by forcing lockfile usage?

[shs96c]

We'd likely want to consolidate on the starlark version of the code, since that's the one that's used by people when they verify the signatures.

[MarconZet]

I agree that it would be the best solution, but it's not simple to implement.

Starlark code runs in analysis phase, this java code runs in execution phase, so it's impossible to do without a minor rewrite of the flow.

[shs96c]

@MarconZet, I'm waiting until you move this out of draft before reviewing. Please LMK when you're ready!

[MarconZet]

@shs96c any progress on the review?

[thomasbao12]

Could we add a description to the PR like:

Summary

This commit introduces lock file version 3 with per-artifact hashing instead of a single global hash. The main purpose is to create "non-conflicting" hashes that allow more granular change
detection in the maven dependency lock files.

Key Changes

1. Lock File Format Change (v2 → v3)

• Before (v2): __INPUT_ARTIFACTS_HASH and __RESOLVED_ARTIFACTS_HASH were single integer values
• After (v3): Both are now dictionaries mapping each artifact coordinate to its individual hash

Example in maven_install.json:
// Old format
"__INPUT_ARTIFACTS_HASH": 1994476565,
"__RESOLVED_ARTIFACTS_HASH": -274973469,

// New format
"__INPUT_ARTIFACTS_HASH": {
"com.google.guava:guava": 733518530,
"junit:junit": -652553691,
...
},
"__RESOLVED_ARTIFACTS_HASH": {
"com.google.guava:guava": -1587873388,
...
}

2. File Renames

• v2_lock_file.bzl → v3_lock_file.bzl
• V2LockFile.java → V3LockFile.java
• V2LockFileTest.java → V3LockFileTest.java

3. Hash Computation Changes (private/rules/v3_lock_file.bzl:53-108)

The new _compute_lock_file_hash_v3 function computes individual hashes per artifact that include:

• The artifact's own info (coordinates, SHA sums)
• The repository it came from
• Hashes of all transitive dependencies (dependency-aware hashing)

4. Input Hash Changes (private/rules/coursier.bzl:334-386)

compute_dependency_inputs_signature now returns a dictionary of per-artifact hashes plus backward-compatible v1/v2 signatures.

5. Command-line Interface Change (pin_dependencies.bzl)

Changed from --input_hash (single value) to --input-hash-path (path to JSON file containing the hash dictionary).

6. Backward Compatibility

The code still supports reading v2 lock files - it checks for v3 first, then falls back to v2, then v1. Users with older lock files will see a message to repin.

Purpose

This per-artifact hashing approach allows the system to detect exactly which artifacts changed, rather than just knowing "something changed." This is useful for incremental updates and
more precise cache invalidation.

[honnix]

We tried this patch and so far it has been working well. There is one thing though. In case of mismatched signature,

rules_jvm_external/private/rules/coursier.bzl

Line 568 in e95b9d7

"%s_install.json contains an invalid signature (expected %s and got %s) and may be corrupted. " % (

prints out a huge single line of artifact shas, for each every artifact. In our case it causes ~2GB of logs.

[MarconZet]

@honnix I changed the code, It should print errors better now

[honnix]

@honnix I changed the code, It should print errors better now

Nice! Thank you. We will take the new patch and try it out.

[vinnybod]

FWIW, We've been using this at Confluent for a month now and it has been working well.

[shs96c]

We'd likely want to consolidate on the starlark version of the code, since that's the one that's used by people when they verify the signatures.

[shs96c]

You should use the same logic that's in Coordinates.asKey() to get a stable key that includes things like the classifier. That's already in coordinates.bzl as to_key

[MarconZet]

hmm, this key has a direct mapping to what appears in the lock file

My idea was, that I don't want things like classifier and packaging, my reason being:

1. It looked ugly in the lock file – when I tried it in my repo, the __INPUT_ARTIFACTS_HASH size duplicated with :sources. It did not provide any information and was just noise.
2. I think that at this level, we don't want to allow a non-conflicting merge.

[shs96c]

Why is the salt needed for artifacts and boms? They should have unique coordinates no matter what.

[MarconZet]

I did this thinking about artifact and excluded_artifact.

In v2 hash, excluding an artifact would change the hash because the has order changed. In group:artifact: HASH notation, excluding an artifact would not change the hash, so we need salt.

The rest is about code design principles, adding salt everywhere is easier then adding salt only to excluded_artifact.

[shs96c]

Let me handle the rebase, and I'll merge this when I've done so.

[shs96c]

Ah! I can't do the rebase. Could you please handle that?

[shs96c]

The test failures look related to this change. The V3LockFileTest is failing.

[MarconZet]

@shs96c I forgot to add some files, It should be ok now

[shs96c]

Let's go! LGTM