[MNG-7802] Fix behaviour of the maven update policy

[jira-importer]

Guillaume Nodet opened MNG-7802 and commented

The update policy can be specified using the -U (force update) or -nsu (no update) options, but those options change the whole repository session policy and override any settings on the repositories.

This means that if -U is set, the resolver will attempt to check already downloaded artifacts.  This is wrong and the behaviour has been inherited from maven 2.x.

What we really wants (and what's implied by the name of the options and docs) is to check for new artifacts / updates, so this mainly affect version resolution and not {}artifact resolution{}.

---

Issue Links:

• MNG-7803 Deprecate -U and -nsu and provide correct options reflecting the real behaviour

• MNG-7801 Clean doc for repositories policies

Remote Links:

• GitHub Pull Request #1144
  

[jira-importer]

Tamas Cservenak commented

I retract my comment above, does not looks good. Here is what happens:

session.UpdatePolicy == IF non-null, session overrides everything, IF null, remoteRepository policy is applied.

So, what happens in maven-core DefaultRepositorySystemSessionFactory is CORRECT:

• -U present? set session to ALWAYS
• -nsu present? set session to NEVER
• otherwise leave it null (leave to remote repository policy to decide)

Other question is like "why are they named like this" and how is this possible:

[cstamas@blondie maven (maven-3.9.x)]$ mvn -U -nsu clean
[INFO] Scanning for projects...
... 

(so what happens now? both options given, but Maven does not nags about nothing)

But this is more for MNG-7803 (change help text and add some validation maybe?)

IMHO, problems for this issue are following:

• default value for update policy = "never" (when value is not set)
• central repo default value for update policy = "never" (as set in Maven)
• "en gros" application of policy (irrelevant is it "global", from session or "local", from remote repo) to both, artifacts and metadata

Default value affects "built in" repository central and release repositories mostly: 

• if we set "never" as today, things like versions:display-plugin-updates and alike (version resolution) will only work with -U, and this is the reason why everyone on earth uses this plugin with -U
• if we set to "daily", it will make Maven perform conditional GETs for release artifacts in central (that are immutable) => wasteful nonsense

Note: "never" policy and versions plugin has been addressed with a hack: mojohaus/versions@82e2450 BUT IMHO this is not a solution, as every other Mojo relying on version resolution is forced to do the same thing!

And here we arrive to 2nd problem: "en gros" application of policy to both, artifacts and metadata.

As discovered during lengthy MRESOLVER-363 (and related mojohaus/versions#959) TLDR; user had corrupted local repo, so "last update" for he's metadata was Jan 1 1970 but Maven did not update (showed new version), as policy for central is "never"! Only -U helps, that not only did pull updates but "fixed" the corruption in local repo as well)

• central with "never" policy causes that central "show me updates" use cases (versions:display-xxx-updates) does not work as expected without -U
• problem was not artifact, but metadata update, and in some sense, resolver was right: did not update, as policy of central is "never"
• resolver distinguish (and have separate exec paths for artifacts and metadata) but applies SAME policy (that configured in remote repository) to BOTH 
• user added repositories MAY or MAY NOT have set policies (users usually set them on snapshot repositories), so similar applies
• ultimately: since Maven 3 (timestamped snapshots supported only) I kinda feel that "update policy" should be applied to metadata ONLY, as if you think about, all the artifacts (release or snapshot) are de facto immutable (in maven2 world, when remote repositories allowed foo-1.0-SNAPSHOT.jar, those were NOT immutable, and had to be updated using usual means, ie. conditional GETs)

 

[jira-importer]

Tamas Cservenak commented

Explanation of artifact immutability: since Maven3 there should be no artifact (POM, JAR, ZIP, ...), addressable by GAV, that once downloaded, can change afterwards on remote repository.

• in release repositories no "redeploy" should be possible
• in snapshot repositories effectively no redeploy can happen (as timestamp and build No will change per deploy)

Artifacts are "input" from users, like dependencies, plugins, parent POMs etc (all addressable as GAV). Once an artifact is downloaded (implies checksum validation, again, according to checksum policy – sadly warn is default), resolver considers locally present file as "valid" file. Due that above (immutability), these files effectively never change and cannot change.

Explanation for metadata: these files are NOT addressable by users, and resolver handles them opaquely to obtain information depending on resolver use case. They ARE stored on same URIs in remote repository and DO change, hence "change detection" in form of policy + conditional GETs (or similar mechanism) is a must.

Consequence related to reproducible builds: due this, if build contains no snapshot version, the artifacts used (any artifact, so plugins, parent POMs, dependencies, etc) in build are ALWAYS same. This obviously does not stand for build containing snapshot versions, as the -SNAPSHOT version in POM may resolve to different timestamped artifacts, hence, without POM change the used artifacts MAY change, hence "moving target".

Note: resolver and maven in this very moment is "prepared" and has "all the infra" to UPDATE artifact files as well (but this effectively never happens). This is basically part of legacy from Maven2 and transition, where non-timestamped snapshots had to be supported by Maven3. We need to decide should we keep this "super power" or simply drop it.

If we drop, solution becomes nearly trivial:

• artifacts are not to be updated at all, they can be "present" or "absent". If present, fine, if absent, fetch them and make them "present".
• metadata should be updated => policy should become applicable to metadata ONLY

In this case "never" (as value and especially as default value) makes no sense, then default value might be "daily".

If we want to stick with this "superpower" to update artifacts, then:

• we may consider to split policies into artifactPolicy and metadataPolicy, but this is not so trivial

[jira-importer]

Garret Wilson commented

I'll try to stay out of this discussion because I don't have time to dig into the deep Maven code internals right now, but I wanted to clarify something said above to put this into context:

As discovered during lengthy MRESOLVER-363 (and related mojohaus/versions#959 TLDR; user had corrupted local repo, so "last update" for he's metadata was Jan 1 1970 but Maven did not update (showed new version), as policy for central is "never"! Only -U helps, that not only did pull updates but "fixed" the corruption in local repo as well)

I disagree that my local repository was "corrupted". Maybe Maven Resolver didn't understand it, but it wasn't corrupted. My resolver-status.properties file was a completely valid properties file with this content:

#Last modified on: Thu Sep 08 15:34:35 PDT 2022
#Thu Sep 08 15:34:35 PDT 2022
central.maven-metadata-central.xml.lastUpdated=1662676475074

The fact that Maven Resolver decided to interpret this as "Jan 1 1970" is a decision Maven Resolver made. This file doesn't say "Jan 1 1970", it says 2022-09-08.

I was told that Maven Resolver recognizes a different property than central.maven-metadata-central.xml.lastUpdated. Still nothing told Maven Resolver "Jan 1 1970". Instead, Maven Resolver took it upon itself to say, "I see that you wanted to indicate when you were updated, but I don't understand it, so I'm never, ever going to update this file."

Additionally we have to ask where central.maven-metadata-central.xml.lastUpdated came from in the first place. Somebody in Versions Maven Plugin #959 said it might have come from some Maven Compatibility Layer. I haven't looked into this to find out the details, but if Maven Resolver doesn't know how to work with data from the Maven Compatibility Layer, then … shouldn't it? My point is that there is no "corruption" here. Everything worked as normal based upon some set of tools, and Maven Resolver just wasn't in the loop on what to expect with what something else wrote.

You and I as humans can look at the file above and figure out what was intended. There is zero ambiguity in the file about what the last updated timestamp should be. The file is 100% machine processable. Shouldn't there be some way to improve Maven Resolver to figure that out as well?

I realize this ticket deals with complicated policy beyond this particular issue, so I'll go back into lurk mode for this ticket and let all of you discuss it. I just wanted to clarify that part. Have a wonderful weekend.

[jira-importer]

Tamas Cservenak commented

Ok, then just for the context: the local repo was "corrupted" in a way that resolver was unable to find the lastUpdated timestamp in otherwise healthy file, in these cases resolver uses timestamp = 1L (1 Jan 1970 + 1ms) to signal this. Still, the effective update policy was "never", so even so "old" file did not trigger metadata update, that resulted in no discovery of new versions (reported as bug).

Reason why it did not find that last updated timestamp is most probably following: Garret used old(er) release of maven-versions-plugin, then switched to new(er) version of same plugin. The old(er) one uses old code and bits from maven-compat (a la Maven2), while new version of plugin was switched over to direct use of resolver. The maven-compat and resolver use same file but different keys to store last updated timestamp (for me due unknown reasons, but I guess is to keep their "update cadence" completely separated).

[jira-importer]

Tamas Cservenak commented

Part of the fix may be this (resolver) change: apache/maven-resolver#297

For this to be complete, things needed:

• Maven3.x in session factory always sets aether.updateCheckManager.updatePolicyScope=all to make Resolver work in "old" way, so this change would not alter Maven3 behaviour (we may or may not allow users to toy with this, so set only if not set by user, but IMHO safest is to just force "all" as for solution to be complete, Maven3 lacks other things, see Maven4)
• Maven4.x does NOT set (so obey resolver default "metadata" OR user set configuration), BUT it should implement following changes as well: CHANGE default update policy value (when value not given) to "daily", and also make sure that "built in" central repo have "daily" policy as well. This way, Maven4 would effectively solve the problem and no "hacks" become needed scattered in various mojos doing version resolution.