JDK 23 support - UnsupportedOperationException: getSubject is supported only if a security manager is allowed

[lpandzic]

Driver version

12.8.1.jre11

SQL Server version

2019

Client Operating System

Rocky Linux

JAVA/JVM version

Eclipse Temurin 23

Problem description

Provide full details of the problem.

Expected behavior

Kerberos authentication should work out of the box with JDK 23 just as it did with JDK 22 and prior versions.

Actual behavior

Application fails to start.

Error message/stack trace

Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: getSubject is supported only if a security manager is allowed
	at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:584)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:571)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:98)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:111)
	at com.infobip.spring.data.common.InfobipSpringDataCommonConfiguration.sqlTemplates(InfobipSpringDataCommonConfiguration.java:21)
	at com.infobip.spring.data.common.InfobipSpringDataCommonConfiguration$$SpringCGLIB$$0.CGLIB$sqlTemplates$0(<generated>)
	at com.infobip.spring.data.common.InfobipSpringDataCommonConfiguration$$SpringCGLIB$$FastClass$$1.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:348)
	at com.infobip.spring.data.common.InfobipSpringDataCommonConfiguration$$SpringCGLIB$$0.sqlTemplates(<generated>)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:146)
	... 124 common frames omitted
Caused by: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
	at java.base/javax.security.auth.Subject.getSubject(Subject.java:347)
	at com.microsoft.sqlserver.jdbc.KerbAuthentication.initAuthInit(KerbAuthentication.java:100)
	at com.microsoft.sqlserver.jdbc.KerbAuthentication.generateClientContext(KerbAuthentication.java:238)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:6596)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:5411)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:5343)
	at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7718)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:4393)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:3830)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:3355)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:3180)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:1963)
	at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1259)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:137)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:360)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:202)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:461)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:550)
	... 135 common frames omitted

Any other details that can be helpful

https://jdk.java.net/23/release-notes#JDK-8296244

While suggested "fix" fixes this issue I assume KerbAuthentication class (and others) should move away from security manager. Especially since it's documented that this fix is going away in the future:

When the Security Manager is removed in a future release, the Subject.getSubject(AccessControlContext) method will be degraded further to throw UnsupportedOperationException unconditionally.

[lpandzic]

Quick follow up - it has been proposed that the workaround fix stops working already in the next version (jdk 24):
https://mail.openjdk.org/pipermail/jdk-dev/2024-September/009420.html
https://openjdk.org/jeps/486
A little bit weird to me that they break the code in a non lts version with a workaround flag and already in next non lts they remove the flag as well. It seems that between the LTS versions it'll look like a completely backwards incompatible change.

[kf-gonzalez2]

Hi, @lpandzic we are aware of the issue and we are working on it

[sj-ganwh]

workaround by adding jvm param: -Djava.security.manager=allow

[paullam328]

pyspark failed due to this error as well:

Python 3.12.4 (v3.12.4:8e8a4baf65, Jun  6 2024, 17:33:18) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
24/10/25 19:16:36 WARN Utils: Your hostname, Pauls-Laptop.local resolves to a loopback address: 127.0.0.1; using 192.168.1.69 instead (on interface en0)
24/10/25 19:16:36 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/10/25 19:16:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/10/25 19:16:36 WARN SparkContext: Another SparkContext is being constructed (or threw an exception in its constructor). This may indicate an error, since only one SparkContext should be running in this JVM (see SPARK-2243). The other SparkContext was created at:
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:501)
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:485)
py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
py4j.Gateway.invoke(Gateway.java:238)
py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
py4j.ClientServerConnection.run(ClientServerConnection.java:106)
java.base/java.lang.Thread.run(Thread.java:1575)
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/python/pyspark/shell.py:74: UserWarning: Failed to initialize Spark session.
  warnings.warn("Failed to initialize Spark session.")
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/python/pyspark/shell.py", line 69, in <module>
    spark = SparkSession._create_shell_session()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/sql/session.py", line 1145, in _create_shell_session
    return SparkSession._getActiveSessionOrCreate()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/sql/session.py", line 1161, in _getActiveSessionOrCreate
    spark = builder.getOrCreate()
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/sql/session.py", line 497, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/context.py", line 515, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/context.py", line 203, in __init__
    self._do_init(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/context.py", line 296, in _do_init
    self._jsc = jsc or self._initialize_context(self._conf._jconf)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/context.py", line 421, in _initialize_context
    return self._jvm.JavaSparkContext(jconf)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1587, in __call__
    return_value = get_return_value(
                   ^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
	at java.base/javax.security.auth.Subject.getSubject(Subject.java:347)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:577)
	at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2416)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2416)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:329)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:501)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:485)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
	at py4j.Gateway.invoke(Gateway.java:238)
	at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
	at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.base/java.lang.Thread.run(Thread.java:1575)

[lilgreenbird]

@paullam328

This is happening because the SecurityManager is deprecated in the later versions of JDK. As mentioned please use the param -Djava.security.manager=allow

[lpandzic]

Not sure what to make of the comment

Hi, @lpandzic we are aware of the issue and we are working on it

followed by many later with the workaround that I linked before already. Is someone working on implementing an actual long-term fix for this issue?

But just to make sure everybody is aware of the other part of my original post since now it has been confirmed - the workaround WILL NOT WORK on jdk 24:
JEP proposed to target JDK 24: 486: Permanently Disable the Security Manager
https://mail.openjdk.org/pipermail/jdk-dev/2024-November/009601.html

Important section from the linked JEP in that mail:

There is a risk that frameworks which rely on the javax.security.auth API still use the deprecated methods in the Subject class, namely doAs and getSubject. We deprecated these methods in Java 17 and 18 because their signatures use deprecated classes of the Security Manager API. We introduced replacements for doAs and getSubject in Java 18. Since getSubject has thrown an UnsupportedOperationException since Java 23, we assume that frameworks have become aware of the deprecation and are working to adopt the replacements, e.g., HADOOP-19212.

Now to catch early the comments that 24 is a non LTS - there's another JEP coming to that release that has a pretty significant interest from wider community so this will likely be a much more sought after jdk release than 23 was:
JEP proposed to target JDK 24: 491: Synchronize Virtual Threads without Pinning
https://mail.openjdk.org/pipermail/jdk-dev/2024-November/009600.html

So it would be really great if this issue would be properly fixed before JDK 24 release which currently is planned for 18th of March, 2025.

[lpandzic]

Thank you for prompt fix!

[JankiPatelM]

spark = SparkSession.builder
.appName("YourAppName")
.config("spark.driver.extraJavaOptions", "-Djava.security.manager=allow")
.config("spark.executor.extraJavaOptions", "-Djava.security.manager=allow")
.getOrCreate()

Use this code to fix that error. I have the same issue but resolved with this code snippet.

[adosib]

Realize this isn't a Spark thread but it's the most ranking result when Googling the error 🤷‍♂ so I'm going to build on the answer from @JankiPatelM

If you're just trying to get a (py)spark shell going and you hit this error:
$ pyspark --driver-java-options -Djava.security.manager=allow