-
Notifications
You must be signed in to change notification settings - Fork 71
Description
This was discovered and fixed yesterday. There was a race condition in how we were resolving the IP address of backend hostnames and then writing those IPs to new configuration files for nginx. This temporarily led to us routing a specific API backend to nowhere, which is quite bad. It would eventually restore itself on its own, but I believe this should now be fixed so this race condition shouldn't be possible.
The race condition was related to a DNS lookup occurring at the exact same time that we were writing a new config file with other DNS changes that had occurred. Given the timings of things, this was more likely to happen for domain name backends that have short TTLs, but then I think we only recently saw this because it was also more likely to occur as we add more and more backends (since there's more simultaneous DNS lookups or changes going on).
Fixed in: NREL/api-umbrella-gatekeeper@dc73bae