[BUG] Installing OpenSearch, the prompt "Unable to read the file root-ca.pem." appears.

[tianxin8206]

What is the bug?
Installing opensearch with helm did not start properly.

java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions]

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Install a default opensearch setup using a helm chart = 2.30.0

What is the expected behavior?
The program starts normally.

What is your host/environment?

• OS: AlmaLinux 9.5 / Kubernetes V1.30
• Helm Charts 2.30.0

Do you have any screenshots?

java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSLContext
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593) ~[?:?]
	at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872) ~[?:?]
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) ~[?:?]
	at java.base/java.lang.Class.forName0(Native Method) ~[?:?]
	at java.base/java.lang.Class.forName(Class.java:534) ~[?:?]
	at java.base/java.lang.Class.forName(Class.java:513) ~[?:?]
	at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:95) ~[netty-handler-4.1.114.Final.jar:4.1.114.Final]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin$4.run(OpenSearchSecuritySSLPlugin.java:218) ~[opensearch-security-2.18.0.0.jar:2.18.0.0]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:213) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) [opensearch-security-2.18.0.0.jar:2.18.0.0]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:523) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:450) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.18.0.jar:2.18.0]
	at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.18.0.jar:2.18.0]
[2025-01-10T07:15:02,827][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-master-1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.18.0.jar:2.18.0]
	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.18.0.jar:2.18.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:805) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
	... 6 more
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
	... 6 more
Caused by: org.opensearch.OpenSearchException: Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions
	at org.opensearch.security.ssl.config.SslCertificatesLoader.resolvePath(SslCertificatesLoader.java:165) ~[?:?]
	at org.opensearch.security.ssl.config.SslCertificatesLoader.loadConfiguration(SslCertificatesLoader.java:85) ~[?:?]
	at org.opensearch.security.ssl.SslSettingsManager.loadConfigurations(SslSettingsManager.java:137) ~[?:?]
	at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:93) ~[?:?]
	at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318) ~[?:?]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:523) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.node.Node.<init>(Node.java:450) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.18.0.jar:2.18.0]
	... 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions]
	at org.opensearch.security.ssl.config.SslCertificatesLoader.resolvePath(SslCertificatesLoader.java:165)
	at org.opensearch.security.ssl.config.SslCertificatesLoader.loadConfiguration(SslCertificatesLoader.java:85)
	at org.opensearch.security.ssl.SslSettingsManager.loadConfigurations(SslSettingsManager.java:137)
	at org.opensearch.security.ssl.SslSettingsManager.buildSslContexts(SslSettingsManager.java:93)
	at org.opensearch.security.ssl.SslSettingsManager.<init>(SslSettingsManager.java:80)
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:249)
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:318)
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:796)
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744)
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:545)
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:197)
	at org.opensearch.node.Node.<init>(Node.java:523)
	at org.opensearch.node.Node.<init>(Node.java:450)
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)

Do you have any additional context?
It looks like opensearch-project/security#4923 .Please confirm that the BUG has been fixed.

[cwperks]

Transferring to helm-charts repo.

[DandyDeveloper]

@tianxin8206 Are you deploying with changes to the values? Can you provide the values you're using?

Specifically, are you disabling the demo installation / demo stuff via the env?

If you do this, it will not run the demo configuration script, resulting in you needing to provide your own certificates for transport SSL between the nodes.

[tianxin8206]

@DandyDeveloper I modified the value of OPENSEARCH_INITIAL_ADMIN_PASSWORD as required in the document, and made no other changes.These are my values.

[scobit]

I have the same problem.

I added only:

OPENSEARCH_INITIAL_ADMIN_PASSWORD

and

storageClass: MyNameHere

Chart version: 2.30.1

Also I have tested charts 2.27.1 and 2.26.1 - similar situation.

Probably it some kind of infra issue, but i have no idea what kind of.

In my case one of three pods is running, but another two have status CrashLoopBackOff.

NAME                          READY   STATUS             RESTARTS     AGE
opensearch-cluster-master-0   0/1     CrashLoopBackOff   2 (6s ago)   55s
opensearch-cluster-master-1   0/1     CrashLoopBackOff   2 (8s ago)   55s
opensearch-cluster-master-2   1/1     Running            0            55s

[DandyDeveloper]

@scobit Can you tell me a little more about your k8s cluster? The fact its other nodes, might mean:

1. You have Opensearch nodes deploying to the same K8s node, which on the backside means they might be trying to share volumes.
2. You have a node that does have a relevant local FS it can mount things against, other nodes might be using a tempfs and the permissions are not allowing access to the CA certificate in question.

Just as an example, I've got a rootless implementation of Minikube running locally and I've used your values and it's working fine:

C02:12:21 dandy@Laptop ~ → kgp -w
NAME                          READY   STATUS    RESTARTS   AGE
opensearch-cluster-master-0   1/1     Running   0          33s
opensearch-cluster-master-1   1/1     Running   0          33s
opensearch-cluster-master-2   1/1     Running   0          33s

Something to try - Override the command in the statefulset with something like comment: ["bash", "-c", "sleep 10000000"]

Then, exec into the failing nodes and see what's in dir: /usr/share/opensearch/config/root-ca.pem

[gurlov]

I have the same problem.
@DandyDeveloper Then, exec into the failing nodes and see what's in dir: /usr/share/opensearch/config/root-ca.pem :

[opensearch@opensearch-cluster-master-0 ~]$ ls -l /usr/share/opensearch/config/
total 40
-rw-rw----. 1 opensearch opensearch 3041 Jan 15 19:14 jvm.options
drwxr-x---. 2 opensearch opensearch 4096 Oct 28 19:44 jvm.options.d
-rw-rw----. 1 opensearch opensearch  514 Jan 15 19:13 log4j2.properties
drwxr-x---. 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-notifications
drwxr-x---. 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-notifications-core
drwxr-x---. 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-observability
drwxr-x---. 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-performance-analyzer
drwxr-x---. 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-reports-scheduler
drwxr-x---. 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-security
-rw-r--r--. 1 opensearch opensearch 1771 Jan 17 14:41 opensearch.yml

[DandyDeveloper]

@gurlov Throw me a complete node log please, looks to me like the demo stuff is failing for some reason.

[gurlov]

@DandyDeveloper with the section plugins: commented in opensearch.yml (This is the default for custom-values.yaml).
When uncommented - visually the errors were the same. But if necessary, I can form them separately and send them.
opensearch.log

[scobit]

@DandyDeveloper in my case, using this code for StatefulSet:

      - command:
        - sh
        - -c
        - |
          #!/usr/bin/env bash
           cp -r /tmp/configfolder/*  /tmp/config/
        - sleep 10000

And running these commands:

k exec -it opensearch-cluster-master-1 -- cat /usr/share/opensearch/config/root-ca.pem
Defaulted container "opensearch" out of: opensearch, configfile (init)
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

k exec -it opensearch-cluster-master-1 -- cat /usr/share/opensearch/config/root-ca.pem
k exec -it opensearch-cluster-master-2 -- cat /usr/share/opensearch/config/root-ca.pem

I figured out that every pod at start has this file. But after some time one of the pods changed status to failed.

NAME                          READY   STATUS             RESTARTS        AGE
opensearch-cluster-master-0   1/1     Running            0               14m
opensearch-cluster-master-1   1/1     Running            0               14m
opensearch-cluster-master-2   0/1     CrashLoopBackOff   7 (2m38s ago)   14m

I am using k8s by RKE v2.10. Above test was without persistence storage.

opensearch-cluster-master-2.log