Investegate incident in NHN 25.02.2025 #37
Description
On the 25.02 a rollout of the agent v2 resulted in a outage in the ror-api.
Root cause:
mongodb stoped replying to queries in a timely manner due to missing indexes on the resourcev2 collection.
Other issues observed:
RabbitMQ had a que that had no consumers making it fill the memory allocated.
The api starts slowly due to ldap servers queried from dns not responding.
Remediation:
- implement indexes on resourcev2 collection #36
- Implement rate limiting/memory monitoring #32
- taint unavailable ldap-servers #33
- investegate possible memory leak in resource v2 endpoint handling #34
- Make all ques registered for micro services autodelete #38
- Create 500 page that links to alternative management documentation #39