Skip to content

Conversation

@challenger71498
Copy link
Contributor

@challenger71498 challenger71498 commented Apr 19, 2025

Details

When the ServiceLoader::load method is invoked with a single argument, it uses the context ClassLoader of current thread(by calling Thread.currentThread.getContextClassLoader()). (For definition, see here.)

Under normal circumstances, this does not cause issues. But there is a problem if you try to:

  • load a class at runtime from a jar file (e.g. call URLClassLoader::loadClass)
    • (the jar file has a dependency with hbase-client)
  • from the parent Application.

In my case, I encountered this problem while developing JDBC wrapper for hbase-shell.

If you load the class from the jar file, the ClassLoader of current thread is normally the class loader used to load the parent application.

  • This "parent" ClassLoader is not adaquate to ServiceLoader at the SaslClientAuthenticationProviders.
  • The ServiceLoader with the "parent" ClassLoader could not access the service info of the jar file(META-INF/services).
  • Therefore, IllegalStateException is being thrown.

I also attached the full stacktrace of the exception just in case.

Stacktrace

org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=2, exceptions:
2025-04-19T06:02:00.404Z, RpcRetryingCaller{globalStartTime=2025-04-19T06:02:00.241Z, pause=100, maxAttempts=2}, java.io.IOException: Call to address=hbase:16020 failed on local exception: java.io.IOException: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.
2025-04-19T06:02:00.518Z, RpcRetryingCaller{globalStartTime=2025-04-19T06:02:00.241Z, pause=100, maxAttempts=2}, java.io.IOException: Call to address=hbase:16020 failed on local exception: java.io.IOException: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.

	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:142)
	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.IOException: Call to address=hbase:16020 failed on local exception: java.io.IOException: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:270)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:395)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:430)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
	at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:116)
	at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:131)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:451)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:94)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:606)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:44040)
	at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:365)
	at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:261)
	at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:62)
	at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:131)
	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:206)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:437)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:405)
	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104)
	... 4 more
Caused by: java.io.IOException: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.
	at org.apache.hadoop.hbase.ipc.IPCUtil.toIOE(IPCUtil.java:190)
	... 17 more
Caused by: java.lang.IllegalStateException: Failed to load SIMPLE, KERBEROS, and DIGEST authentication providers. Classpath is not sane.
	at org.apache.hadoop.hbase.security.provider.BuiltInProviderSelector.configure(BuiltInProviderSelector.java:93)
	at org.apache.hadoop.hbase.security.provider.SaslClientAuthenticationProviders.instantiateSelector(SaslClientAuthenticationProviders.java:114)
	at org.apache.hadoop.hbase.security.provider.SaslClientAuthenticationProviders.instantiate(SaslClientAuthenticationProviders.java:187)
	at org.apache.hadoop.hbase.security.provider.SaslClientAuthenticationProviders.getInstance(SaslClientAuthenticationProviders.java:76)
	at org.apache.hadoop.hbase.ipc.RpcConnection.<init>(RpcConnection.java:138)
	at org.apache.hadoop.hbase.ipc.NettyRpcConnection.<init>(NettyRpcConnection.java:108)
	at org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:91)
	at org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:47)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.lambda$getConnection$0(AbstractRpcClient.java:368)
	at org.apache.hadoop.hbase.util.PoolMap.createResource(PoolMap.java:127)
	at org.apache.hadoop.hbase.util.PoolMap$RoundRobinPool.getOrCreate(PoolMap.java:211)
	at org.apache.hadoop.hbase.util.PoolMap.getOrCreate(PoolMap.java:68)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:368)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:448)
	... 16 more

I could reproduce this issue with the repository which I made by myself in here. You could also look into it for more information.

Suggestion

As @AbilashR mentioned at the issue, this can be fixed by adding ClassLoader of current class explicitly to the parameter.

I have run the test locally and had no issues. Please let me know if there are more tests I should run.

I also tried to add a test case for this, but gave up since reproducing the issue is quite complicated.

Any opinions & suggestions are appriciated. Also, I don't have much experiences in Java, so feel free to make any corrections.

Thank you :)

@challenger71498
Copy link
Contributor Author

challenger71498 commented Apr 19, 2025

Thankfully there is a workaround for this issue.

By setting extra providers option to built-in providers, you may bypass the issue and load the built-in providers.

For example:

<property>
    <name>hbase.client.sasl.provider.extras</name>
    <value>org.apache.hadoop.hbase.security.provider.SimpleSaslClientAuthenticationProvider,org.apache.hadoop.hbase.security.provider.GssSaslClientAuthenticationProvider,org.apache.hadoop.hbase.security.provider.DigestSaslClientAuthenticationProvider</value>
</property>

See here for more context.

@challenger71498 challenger71498 changed the title HBASE-25338 use ClassLoader of current class instead of thread context ClassLoader HBASE-25338 Use ClassLoader of current class instead of thread context ClassLoader Apr 19, 2025
@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache9
Copy link
Contributor

Apache9 commented Apr 21, 2025

Please add a new commit which modifies pom.xml so we can trigger tests for the whole project. We can remove it while merging.

Thanks.

@challenger71498
Copy link
Contributor Author

Please add a new commit which modifies pom.xml so we can trigger tests for the whole project. We can remove it while merging.

Thanks.

You mean the root pom.xml right? No problem, just edited it and pushed it.

@challenger71498
Copy link
Contributor Author

Just in case, also attaching the test result of my test project mentioned above to be sure that the edit resolves the issue.

I made a custom build of hbase-client from branch-2.5, using hadoop 2, PR applied.

Image 1 shows that the custom build resolves the problem.
Image 2 shows the current behavior of 2.6.2 in contrast, which fails.

image image

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache9
Copy link
Contributor

Apache9 commented Apr 24, 2025

Seems OK, please revert the commit which modifies pom.xml, I will merge it later.

Thanks @challenger71498

@challenger71498
Copy link
Contributor Author

@Apache9 Will this PR be backported to 2.x? Since the master branch is for 4.0 (which hasn’t been released yet), and I am currently using version 2.6.2, it would be great if this PR could be backported.

@Apache9 Apache9 merged commit bd30367 into apache:master Apr 25, 2025
1 check was pending
Apache9 pushed a commit that referenced this pull request Apr 25, 2025
…t ClassLoader (#6918)

Signed-off-by: Duo Zhang <[email protected]>
(cherry picked from commit bd30367)
Apache9 pushed a commit that referenced this pull request Apr 25, 2025
…t ClassLoader (#6918)

Signed-off-by: Duo Zhang <[email protected]>
(cherry picked from commit bd30367)
Apache9 pushed a commit that referenced this pull request Apr 25, 2025
…t ClassLoader (#6918)

Signed-off-by: Duo Zhang <[email protected]>
(cherry picked from commit bd30367)
Apache9 pushed a commit that referenced this pull request Apr 25, 2025
…t ClassLoader (#6918)

Signed-off-by: Duo Zhang <[email protected]>
(cherry picked from commit bd30367)
@Apache9
Copy link
Contributor

Apache9 commented Apr 25, 2025

@challenger71498 Could you please reply on the jira so I can assign the jira to you?

I've cherry-picked the patch to all active branches.

Thanks for contributing!

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 57s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
_ master Compile Tests _
+0 🆗 mvndep 1m 15s Maven dependency ordering for branch
+1 💚 mvninstall 5m 17s master passed
+1 💚 compile 10m 46s master passed
+1 💚 checkstyle 1m 21s master passed
+1 💚 spotbugs 11m 7s master passed
+1 💚 spotless 1m 20s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 16s Maven dependency ordering for patch
+1 💚 mvninstall 3m 39s the patch passed
+1 💚 compile 11m 39s the patch passed
+1 💚 javac 11m 39s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 2m 4s the patch passed
+1 💚 xmllint 0m 1s No new issues.
+1 💚 spotbugs 11m 18s the patch passed
+1 💚 hadoopcheck 18m 15s Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚 spotless 0m 59s patch has no errors when running spotless:check.
_ Other Tests _
+1 💚 asflicense 0m 23s The patch does not generate ASF License warnings.
90m 52s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6918/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #6918
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless xmllint
uname Linux 4831baf87c70 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5dfa9b6
Default Java Eclipse Adoptium-17.0.11+9
Max. process+thread count 192 (vs. ulimit of 30000)
modules C: hbase-client . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6918/4/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3 xmllint=20913
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@challenger71498
Copy link
Contributor Author

Sure, I just requested a Jira account by self-service portal. I will leave a comment after getting one :)

@challenger71498
Copy link
Contributor Author

@Apache9 Just added the comment at the issue here.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 59s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 1m 14s Maven dependency ordering for branch
+1 💚 mvninstall 5m 16s master passed
+1 💚 compile 3m 3s master passed
+1 💚 javadoc 3m 24s master passed
+1 💚 shadedjars 7m 47s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 16s Maven dependency ordering for patch
+1 💚 mvninstall 4m 6s the patch passed
+1 💚 compile 3m 27s the patch passed
+1 💚 javac 3m 27s the patch passed
+1 💚 javadoc 2m 50s the patch passed
+1 💚 shadedjars 7m 48s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
-1 ❌ unit 298m 19s /patch-unit-root.txt root in the patch failed.
345m 52s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6918/4/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR #6918
Optional Tests javac javadoc unit compile shadedjars
uname Linux bfc5df56c70a 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5dfa9b6
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6918/4/testReport/
Max. process+thread count 8290 (vs. ulimit of 30000)
modules C: hbase-client . U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6918/4/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

mokai87 pushed a commit to mokai87/hbase that referenced this pull request Aug 7, 2025
…t ClassLoader (apache#6918)

Signed-off-by: Duo Zhang <[email protected]>
(cherry picked from commit bd30367)
sanjeet006py pushed a commit to sanjeet006py/hbase that referenced this pull request Sep 26, 2025
…t ClassLoader (apache#6918)

Signed-off-by: Duo Zhang <[email protected]>
(cherry picked from commit bd30367)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants