API responses with very long HTTP headers may return 502 errors #461
Description
If an API backend returns a response with more than 4KB of HTTP headers, the end user receives a 502 bad gateway error instead of the actual API response.
I think this is fairly rare that an API backend returns HTTP headers this large, however we did see this recently crop up on some NREL APIs, but only under very specific circumstances. The actual API itself wasn't returning headers this big, but some other firewalls in between were adding some very large Set-Cookie headers on the HTTP response if the user making the request had sent in very large cookies on the initial API request. These large cookies were a result of large cookies being set on the *.nrel.gov domain for internal NREL-employees for unrelated systems (but since they were set at the nrel.gov-level, they were getting included on the developer.nrel.gov API requests made in the browser). So basically, this issue was only affecting certain NREL employees (that happened to have the large cookies set in their browser), and only when they made the API requests in their browser (where these large cookies existed).
The reason for this error is that our nginx layer was throwing this error:
upstream sent too big header while reading response header from upstream
This is related to the nginx_buffer_size setting, which controls the maximum size of the response headers. This setting defaults to either 4KB or 8KB, depending on platform, and it looks like our system defaults to 4KB.
It's possible this default value changed from 8KB to 4KB in some recent server migrations, but I think it's perhaps been 4KB all along, and this is just a rare enough issue that we're only now hearing some users report this. In searching through our recent logs, the only occurrences of this error that I see are the NREL employees hitting a specific API, so again, I think this is pretty rare.
So to solve this, I think we just need to adjust this setting, and probably 8KB should suffice. The specific occurrences I have seen are all hovering right around 4KB, so I think bumping this up to 8KB should be plenty.