large results cause 500 Internal Server Error

[danstoner]

Depending on deployment architecture, may also cause 502 Bad Gateway from a frontend proxy server, etc.

[danstoner]

An end-user reported issue thus:

I have a weird question about the iDigBio API. When I use the first of the two below links, I get an internal server error or sometimes a bad gateway error. When I use the second, it works. As far as I can tell, the only difference between the two links is that I replaced "Mugil" with "Gorilla". These are both genera for which there are records in iDigBio. Any ideas why this is?

https://search.idigbio.org/v2/search/records/?rq={%22genus%22:%20%22Mugil%22}&limit=100000

https://search.idigbio.org/v2/search/records/?rq={%22genus%22:%20%22Gorilla%22}&limit=100000

Thanks much!

[danstoner]

The API has some rough edges and they might need to stay rough for a while. I do not want the API to return 502 Bad Gateway in this case.

We also noticed some funkiness with our backend Elasticsearch service over the past few weeks (brief periods of unavailability) but have not nailed down the cause of that. I don't think that is related to your report but just mention it in case you start getting weird responses (or no responses).

In your reported case, one major difference is that genus Mugil returns many records, whereas I would not expect Gorilla to return very many.

The 500 Internal Server Error comes from the backend error condition bubbling its way out to you. The 502 Bad Gateway is from the frontend proxy server, which maybe means the backend crashed instead of issuing the 500.

The response size definitely seems to affect the behavior.

dan@D810:~$ http --headers --timeout 60 'https://search.idigbio.org/v2/search/records/?rq={%22genus%22:%20%22Mugil%22}&limit=100000'
HTTP/1.0 502 Bad Gateway
Cache-Control: no-cache
Connection: close
Content-Type: text/html

dan@D810:~$ http --headers --timeout 60 'https://search.idigbio.org/v2/search/records/?rq={%22genus%22:%20%22Mugil%22}&limit=10000'
HTTP/1.1 500 Internal Server Error
Cache-Control: public, max-age=300
Content-Length: 33
Content-Type: application/json; charset=utf-8
Date: Sun, 01 Apr 2018 14:33:39 GMT
Vary: Accept-Encoding, Origin

dan@D810:~$ http --headers --timeout 60 'https://search.idigbio.org/v2/search/records/?rq={%22genus%22:%20%22Mugil%22}&limit=1000'
HTTP/1.1 200 OK
Cache-Control: public, max-age=300
Content-Length: 91756921
Content-Type: application/json; charset=utf-8
Date: Sun, 01 Apr 2018 14:36:41 GMT
Last-Modified: Wed, 21 Mar 2018 15:57:28 GMT
Vary: Accept-Encoding, Origin





dan@D810:~$ http --headers --timeout 60 'https://search.idigbio.org/v2/search/records/?rq={%22genus%22:%20%22Gorilla%22}&limit=100000'
HTTP/1.1 200 OK
Cache-Control: public, max-age=300
Content-Length: 10586442
Content-Type: application/json; charset=utf-8
Date: Sun, 01 Apr 2018 14:22:56 GMT
Last-Modified: Wed, 21 Mar 2018 15:57:28 GMT
Vary: Accept-Encoding, Origin

My next question is "what are you actually trying to get out of API?" One way to reduce the size of the result (and improve chances of success) is to use the "fields" parameter to return only those fields you actually need, rather than every field that we store. If I limit the results to a single field "uuid" the large query works ok.

dan@D810:~$ http --headers --timeout 60 'https://search.idigbio.org/v2/search/records/?rq={%22genus%22:%20%22Mugil%22}&fields=["uuid"]&limit=100000'
HTTP/1.1 200 OK
Cache-Control: public, max-age=300
Content-Length: 859884
Content-Type: application/json; charset=utf-8
Date: Sun, 01 Apr 2018 14:41:07 GMT
Last-Modified: Wed, 21 Mar 2018 15:57:28 GMT
Vary: Accept-Encoding, Origin

Seems to also work with 5 fields, probably more.

[danstoner]

TODO: Add the "fields" tip prominently in the wiki.

[danstoner]

Here are server-side logs when running query the big query
/v2/search/records/?rq={"genus":"Mugil"}&limit=100000:

2018-04-02T13:16:12.882Z - info: 200.144.120.56 - "POST /v2/search/records/ HTTP/1.1" 200 85 - 11.730 ms
2018-04-02T13:16:13.008Z - error: uncaughtException: Invalid string length date=Mon Apr 02 2018 13:16:13 GMT+0000 (America), pid=251, uid=33, gid=33, cwd=/var/www, execPath=/usr/bin/node, version=v6.11.5, argv=[/usr/bin/node, /var/www/index.js], rss=706891776, heapTotal=727814144, heapUsed=589580656, external=4438743, loadavg=[0.2529296875, 0.14453125, 0.09912109375], uptime=868042, trace=[column=34, file=/var/www/node_modules/elasticsearch/src/lib/connectors/http.js, function=null, line=180, method=null, native=false, column=13, file=events.js, function=emitOne, line=96, method=null, native=false, column=7, file=events.js, function=IncomingMessage.emit, line=188, method=emit, native=false, column=18, file=_stream_readable.js, function=readableAddChunk, line=176, method=null, native=false, column=10, file=_stream_readable.js, function=IncomingMessage.Readable.push, line=134, method=push, native=false, column=22, file=_http_common.js, function=HTTPParser.parserOnBody, line=123, method=parserOnBody, native=false, column=20, file=_http_client.js, function=Socket.socketOnData, line=363, method=socketOnData, native=false, column=13, file=events.js, function=emitOne, line=96, method=null, native=false, column=7, file=events.js, function=Socket.emit, line=188, method=emit, native=false, column=18, file=_stream_readable.js, function=readableAddChunk, line=176, method=null, native=false, column=10, file=_stream_readable.js, function=Socket.Readable.push, line=134, method=push, native=false, column=20, file=net.js, function=TCP.onread, line=547, method=onread, native=false], stack=[RangeError: Invalid string length,     at IncomingMessage.<anonymous> (/var/www/node_modules/elasticsearch/src/lib/connectors/http.js:180:34),     at emitOne (events.js:96:13),     at IncomingMessage.emit (events.js:188:7),     at readableAddChunk (_stream_readable.js:176:18),     at IncomingMessage.Readable.push (_stream_readable.js:134:10),     at HTTPParser.parserOnBody (_http_common.js:123:22),     at Socket.socketOnData (_http_client.js:363:20),     at emitOne (events.js:96:13),     at Socket.emit (events.js:188:7),     at readableAddChunk (_stream_readable.js:176:18),     at Socket.Readable.push (_stream_readable.js:134:10),     at TCP.onread (net.js:547:20)]
2018-04-02T13:16:13.051Z - warn: Server(251) died.

[wilsotc]

This issue has been resolved. It was caused by a nested aggregation in ES

[nrejac]

I looked at this issue yesterday, the below query still returns with a 500 error:

https://search.idigbio.org/v2/search/records/?rq={%22genus%22:%20%22Mugil%22}&limit=100000

[wilsotc]

Confused this with the map issue. It's probably the same issue. I'll take a look today.

[jhpoelen]

@wilsotc @nrejack hi! Today, I noticed that the iDigBio api stops serving pages after 100k items.

with

https://search.idigbio.org/v2/search/records/?rq=%7B%22recordset%22%3A+%22eaa5f19e-ff6f-4d09-8b55-4a6810e77a6c%22%7D&limit=100&offset=100000

iDigBio says 502 internal server error,

however, to a page request prior to that e.g.,

https://search.idigbio.org/v2/search/records/?rq=%7B%22recordset%22%3A+%22eaa5f19e-ff6f-4d09-8b55-4a6810e77a6c%22%7D&limit=100&offset=99900

iDigBio says: 200 ok, and services a 100 items.

This limit is indicative of some elastic search client/server configuration.

I found a possibly related issue at https://stackoverflow.com/questions/41655913/how-do-i-retrieve-more-than-10000-results-events-in-elastic-search .

[jhpoelen]

btw, restricting the fields doesn't seem to matter

e.g.,

https://search.idigbio.org/v2/search/records/?rq={%22recordset%22%3A+%22eaa5f19e-ff6f-4d09-8b55-4a6810e77a6c%22}&fields=[%22uuid%22]&limit=100&offset=100000

the iDigBio API still says, 502 internal server error.

[wilsotc]

We have a download API that handles >100K records. That being said, I'm currently working on allowing the use of offset to retrieve unlimited records via the search API.

[jhpoelen]

@wilsotc thanks for the update! Hoping to use the search api instead of the download api because the json formatted results work well for my use cases: crawling the idigbio registries and building static websites.