Failed to connect to database "st2" @ "stackstorm-ha" as user "admin": No replica set members found yet #143
Description
Hi All,
We are getting issue while installing the StackStorm ha on Kubernetes. No other pod than st2client, able to connect to mangodb
Error:
kubectl logs stackstorm-ha-1595399628-st2actionrunner-5768ddf56c-cnc57
2020-07-22 08:17:18,737 DEBUG [-] Using Python: 3.6.9 (/opt/stackstorm/st2/bin/python)
2020-07-22 08:17:18,737 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2020-07-22 08:17:18,738 DEBUG [-] Using logging config: /etc/st2/logging.actionrunner.conf
2020-07-22 08:17:18,767 INFO [-] Connecting to database "st2" @ "stackstorm-ha-1595399628-mongodb-ha-0.stackstorm-ha-1595399628-mongodb-ha:27017,stackstorm-ha-1595399628-mongodb-ha-1.stackstorm-ha-1595399628-mongodb-ha:27017,stackstorm-ha-1595399628-mongodb-ha-2.stackstorm-ha-1595399628-mongodb-ha:27017 (replica set)" as user "admin".
2020-07-22 08:17:21,780 ERROR [-] Failed to connect to database "st2" @ "stackstorm-ha-1595399628-mongodb-ha-0.stackstorm-ha-1595399628-mongodb-ha:27017,stackstorm-ha-1595399628-mongodb-ha-1.stackstorm-ha-1595399628-mongodb-ha:27017,stackstorm-ha-1595399628-mongodb-ha-2.stackstorm-ha-1595399628-mongodb-ha:27017 (replica set)" as user "admin": No replica set members found yet
2020-07-22 08:17:21,781 ERROR [-] (PID=1) Worker quit due to exception.
Traceback (most recent call last):
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2actions/cmd/actionrunner.py", line 95, in main
_setup()
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2actions/cmd/actionrunner.py", line 56, in _setup
register_signal_handlers=True, service_registry=True, capabilities=capabilities)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/service_setup.py", line 160, in setup
db_setup()
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/database_setup.py", line 56, in db_setup
connection = db_init.db_setup_with_retry(**db_cfg)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/persistence/db_init.py", line 75, in db_setup_with_retry
ssl_match_hostname=ssl_match_hostname)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/persistence/db_init.py", line 58, in db_func_with_retry
return retrying_obj.call(db_func, *args, **kwargs)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 206, in call
return attempt.get(self._wrap_exception)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "/opt/stackstorm/st2/lib/python3.6/site-packages/six.py", line 696, in reraise
raise value
File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/init.py", line 169, in db_setup
ssl_match_hostname=ssl_match_hostname)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/init.py", line 151, in _db_connect
raise e
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/init.py", line 144, in _db_connect
connection.admin.command('ismaster')
File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/database.py", line 730, in command
read_preference, session) as (sock_info, slave_ok):
File "/usr/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1298, in _socket_for_reads
server = self._select_server(read_preference, session)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1253, in _select_server
server = topology.select_server(server_selector)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 235, in select_server
address))
File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 193, in select_servers
selector, server_timeout, address)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 209, in _select_servers_loop
Using Kubernetes version 1.18.6
SS-ha version - 0.31.0
Also, we have checked that mongodb-replicaset is also downgraded to 3.12.0 and still it not working. Please help
Thanks in advance