[MNG-6275] ServiceLoaderFactory can't find implementations via ClassRealm

[jira-importer]

Robert Scholte opened MNG-6275 and commented

Spotted this issue via MANTRUN-200. The reason is that in the DefaultClassRealmManager a new realm is created where the parent classLoader is null. This implies that the bootstrap classloader is used as parent.
With Java8 nashorn has become an extension and is not part of the bootstrap classloader anymore.
It is kind of strange that we want the bootstrap classloader here, it makes more sense if the system classloader is used (but with Java 7 and older versions of Java this was not an issue, both worked fine).

---

Issue Links:

• MANTRUN-200 Scriptdef tasks fail to load when running on Java9

• MNG-6371 Use ServiceLoader mechanism in JDK9 in maven plugins

0 votes, 11 watchers

[jira-importer]

Robert Scholte commented

Fixed in f047ea143766fd22ae42040e6805bef287f3cc3e

[jira-importer]

Igor Fedorenko commented

I don't have time to do proper analysis of this, but this change will likely cause problems in m2e, plugin testing, jenkins and other embedded usecases, where contents of system classloader is generally unknown and can contain classes that either collide or confuse plugins in some other ways (e.g. random/unrelated components visible to plugins in only some environments, which will be very hard to troubleshoot).

PS: didn't we agree to get all core changes reviewed and +1'ed by at least one other developer before pushing to master?

[jira-importer]

Robert Scholte commented

Hi Igor,

moving this to dev-list.
I've asked an explanation of the Java developers team. They confirm that
they've made a more clear separation of boot-, system- and
application-classloader. They wondered why we put "null" there in the
first place, because in general the boot classloader is not enough.
I've tested if and this change would be the required fix for MANTRUN-200.
I haven't found any failing tests yet.
If this causes issues for other frameworks we need to have a look what
should be changed combined with the impact. Were they relying on a bug in
Maven. Based on the analysis so far this should be the proper fix.
Can you somehow try to verify first if this is really an issue. I'm not
sure if this should be reverted because of potential issues, because in
the end every change could result in new issues.

I'll try to prepare new Maven installation locally and run other Maven
subprojects as well.

thanks,
Robert

ps Stephen gave me the +1 for merging.

On Thu, 24 Aug 2017 14:30:00 +0200, Igor Fedorenko (JIRA)

[jira-importer]

Robert Scholte commented

Reopening: Maven Integration Tests fails with the following message

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-dependency-plugin:3.0.0:resolve-plugins
(get-all-plugins-deps) on project maven-it-boostrap: Execution
get-all-plugins-deps of goal
org.apache.maven.plugins:maven-dependency-plugin:3.0.0:resolve-plugins
failed: Unable to load the mojo 'resolve-plugins' (or one of its required
components) from the plugin
'org.apache.maven.plugins:maven-dependency-plugin:3.0.0':
com.google.inject.ProvisionException: Unable to provision, see the
following errors:
[ERROR]
[ERROR] 1) No implementation for
org.apache.maven.artifact.handler.manager.ArtifactHandlerManager was bound.
[ERROR]   while locating
org.apache.maven.plugins.dependency.resolvers.ResolvePluginsMojo
[ERROR]   at
ClassRealm[plugin>org.apache.maven.plugins:maven-dependency-plugin:3.0.0,
parent: java.net.URLClassLoader@77f03bb1] (via modules:
org.eclipse.sisu.wire.WireModule ->
org.eclipse.sisu.plexus.PlexusBindingModule)
[ERROR]   while locating org.apache.maven.plugin.Mojo annotated with
@com.google.inject.name.Named(value=org.apache.maven.plugins:maven-dependency-plugin:3.0.0:resolve-plugins)
[ERROR]
[ERROR] 1 error
[ERROR]       role: org.apache.maven.plugin.Mojo
[ERROR]   roleHint:
org.apache.maven.plugins:maven-dependency-plugin:3.0.0:resolve-plugins
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
goal org.apache.maven.plugins:maven-dependency-plugin:3.0.0:resolve-plugins
(get-all-plugins-deps) on project maven-it-boostrap: Execution
get-all-plugins-deps of goal
org.apache.maven.plugins:maven-dependency-plugin:3.0.0:resolve-plugins
failed: Unable to load the mojo 'resolve-plugins' (or one of its required
components) from the plugin
'org.apache.maven.plugins:maven-dependency-plugin:3.0.0'
 at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
 at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
 at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
 at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
 at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
 at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
 at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
 at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
 at org.apache.maven.cli.MavenCli.execute(MavenCli.java:955)
 at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:290)
 at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:245)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.maven.it.Embedded3xLauncher.run(Embedded3xLauncher.java:247)
 at org.apache.maven.it.Verifier.executeGoals(Verifier.java:1354)
 at org.apache.maven.it.Verifier.executeGoal(Verifier.java:1254)
 at org.apache.maven.it.Verifier.executeGoal(Verifier.java:1248)
 at
org.apache.maven.it.MavenITBootstrapTest.testBootstrap(MavenITBootstrapTest.java:54)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at junit.framework.TestCase.runTest(TestCase.java:168)
 at
org.apache.maven.it.AbstractMavenIntegrationTestCase.runTest(AbstractMavenIntegrationTestCase.java:222)
 at junit.framework.TestCase.runBare(TestCase.java:134)
 at junit.framework.TestResult$1.protect(TestResult.java:110)
 at junit.framework.TestResult.runProtected(TestResult.java:128)
 at junit.framework.TestResult.run(TestResult.java:113)
 at junit.framework.TestCase.run(TestCase.java:124)
 at junit.framework.TestSuite.runTest(TestSuite.java:243)
 at junit.framework.TestSuite.run(TestSuite.java:238)
 at junit.framework.TestSuite.runTest(TestSuite.java:243)
 at junit.framework.TestSuite.run(TestSuite.java:238)
 at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
 at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
 at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
 at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
 at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
 at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
 at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
 at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution
get-all-plugins-deps of goal
org.apache.maven.plugins:maven-dependency-plugin:3.0.0:resolve-plugins
failed: Unable to load the mojo 'resolve-plugins' (or one of its required
components) from the plugin
'org.apache.maven.plugins:maven-dependency-plugin:3.0.0'
 at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:156)
 at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
 ... 44 more
Caused by: org.apache.maven.plugin.PluginContainerException: Unable to load
the mojo 'resolve-plugins' (or one of its required components) from the
plugin 'org.apache.maven.plugins:maven-dependency-plugin:3.0.0'
 at
org.apache.maven.plugin.internal.DefaultMavenPluginManager.getConfiguredMojo(DefaultMavenPluginManager.java:553)
 at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:124)
 ... 45 more
Caused by:
org.codehaus.plexus.component.repository.exception.ComponentLookupException:
com.google.inject.ProvisionException: Unable to provision, see the
following errors:
1) No implementation for
org.apache.maven.artifact.handler.manager.ArtifactHandlerManager was bound.
  while locating
org.apache.maven.plugins.dependency.resolvers.ResolvePluginsMojo
  at
ClassRealm[plugin>org.apache.maven.plugins:maven-dependency-plugin:3.0.0,
parent: java.net.URLClassLoader@77f03bb1] (via modules:
org.eclipse.sisu.wire.WireModule ->
org.eclipse.sisu.plexus.PlexusBindingModule)
  while locating org.apache.maven.plugin.Mojo annotated with
@com.google.inject.name.Named(value=org.apache.maven.plugins:maven-dependency-plugin:3.0.0:resolve-plugins)
1 error
      role: org.apache.maven.plugin.Mojo
  roleHint:
org.apache.maven.plugins:maven-dependency-plugin:3.0.0:resolve-plugins
 at
org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:267)
 at
org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:255)
 at
org.apache.maven.plugin.internal.DefaultMavenPluginManager.getConfiguredMojo(DefaultMavenPluginManager.java:519)
 ... 46 more
Caused by: com.google.inject.ProvisionException: Unable to provision, see
the following errors:
1) No implementation for
org.apache.maven.artifact.handler.manager.ArtifactHandlerManager was bound.
  while locating
org.apache.maven.plugins.dependency.resolvers.ResolvePluginsMojo
  at
ClassRealm[plugin>org.apache.maven.plugins:maven-dependency-plugin:3.0.0,
parent: java.net.URLClassLoader@77f03bb1] (via modules:
org.eclipse.sisu.wire.WireModule ->
org.eclipse.sisu.plexus.PlexusBindingModule)
  while locating org.apache.maven.plugin.Mojo annotated with
@com.google.inject.name.Named(value=org.apache.maven.plugins:maven-dependency-plugin:3.0.0:resolve-plugins)
1 error
 at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1025)
 at org.eclipse.sisu.inject.Guice4$1.get(Guice4.java:162)
 at org.eclipse.sisu.inject.LazyBeanEntry.getValue(LazyBeanEntry.java:81)
 at org.eclipse.sisu.plexus.LazyPlexusBean.getValue(LazyPlexusBean.java:51)
 at
org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:263)
 ... 48 more

[jira-importer]

Stephen Connolly commented

I believe I have the fix... 39004f6

[jira-importer]

Stephen Connolly commented

https://builds.apache.org/view/M-R/view/Maven/job/maven-3.x-jenkinsfile/job/mng-6275/ will have the test status but a local run of the integration tests reveals this change at least gets past testBootstrap, so likely is good

[jira-importer]

Stephen Connolly commented

Pulling back in scope for 3.5.1 as we look to have a candidate viable fix: https://builds.apache.org/blue/organizations/jenkins/maven-3.x-jenkinsfile/activity?branch=mng-6275 https://github.com/apache/maven/compare/mng-6275

[jira-importer]

Stephen Connolly commented

Merged in master toward 3.5.1

[jira-importer]

Sylwester Lachiewicz commented

I think we fixed this problem with 3.5.2 - I can see both commits in master.

[jira-importer]

Robert Scholte commented

No, it has been reverted

[jira-importer]

Chapman Flack commented

Hi,

By following a link from the maven-antrun-plugin, I am under the impression that this issue might be the underlying reason I am seeing this error with a build that includes a <script> in an <antrun>:

Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (pg_config to pgsql.properties) on project pljava-so: An Ant BuildException has occured: Unable to create javax script engine for javascript

Does that sound right? If so, what's the outlook for a resolution?

Thanks!

[jira-importer]

Matthias Hild commented

This issue is rightfully marked as critical and blocks migration to Java >= 9 for any plugins relying on ServiceLoader logic.

Could you please comment on where you see the problem with the August 2017 commit that was trying to address the issue:

27a2bda#diff-1cb44e9c06f435b494a39ea8bbaed863

This commit uses the class-loader of ClassRealmManager.class as the parent class-loader when creating a new ClassRealm. This commit was reverted in October 2017:

cae779e#diff-1cb44e9c06f435b494a39ea8bbaed863

I recompiled 3.6.3-SNAPSHOT with this fix in place and, for example, the JS ScriptEngine (Nashorn) now loads as expected.

Moreover, the logic of this fix seems at first glance correct.

Many thanks for your great work!!!

M.

PS: With the risk of restating what is already obvious the the committers, here a brief problem statement for the uninitiated: java.util.ServiceLoader was heavily revised in Java 9 (see comments in code) and its method public Iterator<S> iterator() now delegates to an iterator constructed of the inner classes ModuleServicesLookupIterator and LazyClassPathLookupIterator.

In the following code snippet from  ModuleServicesLookupIterator, ``DefaultClassRealmManager is the currentLoader. As its .parent is null in the released versions of Maven, iteratorFor(null) uses the BootLoader's ServiceCatalog where the requested services cannot be not found (as pointed out in earlier posts). 

 

@Override
public boolean hasNext() {
   ...
     // get next provider to load
     while (!iterator.hasNext()) {
       if (currentLoader == null) {
        return false;
      } else {
         currentLoader = currentLoader.getParent();
         iterator = iteratorFor(currentLoader);
       }

    }
  ...

}

 

[jira-importer]

Guillaume Nodet commented

I think a better solution was provided with 39004f6