File uploads are buffered in memory

[abyrd]

We are seeing slowdowns that may be caused by insufficient heap space. Investigating this, we find that there is a bug/feature in Spark web framework that causes all uploads to be buffered in memory. It has been fixed in the development version of Spark, but not yet released.
perwendel/spark#891

Whether or not this is the cause of our slowdowns, it should be addressed. We don't want to be needlessly buffering 10 copies of the Netherlands feed in memory at any given time.

One option is to await the next release of Spark. Another is to not use Spark, assuming all the functionality we need is in the underlying Jetty library.

[abyrd]

We also observe that on the server that experienced the slowdown, memory consumption was very high after the uploads (80% of 6GB of heap) even when nothing was happening. This consumption fell rapidly after additional activity apparently triggered garbage collection, but still, something was using 80% of heap at one point.

[abyrd]

@landonreed and I tore this apart and found several places where uploaded data were being buffered. First, the file is uploaded as multipart form data in a POST request. The javax.servlet.http.HttpServletRequest#getPart implementation in Jetty (which underlies Spark) seems like it must buffer the parts - it allows fetching and re-fetching the parts out of order by name, which would almost have to buffer them. Then Spark itself wraps the request object, overriding the getInputStream method to cache the entire contents of the request body. So it's entirely possible that Spark first caches the entire uploaded file, then the single part it contains (the file) is again cached.

If you don't call any of the multipart retrieval methods, the input stream remains uncached. So you can immediately call raw() on the Spark request, cast the resulting Java Servlet interface to the internal Spark wrapper class, and call the method to extract the actual Jetty request object from inside all these layers. You can then get the uncached inputStream for the body of the request, and then use Guava ByteStreams to copy it into a file with only a 4k buffer.

Initial tests indicate that this brings memory consumption down from several GB to something like 25-50MB.

[abyrd]

This seems to be resolved now, seeing much lower memory use.