Improving ext storage with streaming

[PVince81]

Currently the performance of external storage is bad because the file must first be downloaded into a temporary file, then fopen() is called on that file and the handle is returned. This means that downloading a 4 GB file from an ext storage will first create a 4 GB temporary file which is then passed to the client as download.

In some cases like downloading, mimetype scanning (if we keep it), antivirus app, etc we are only interested in getting either the first bytes of the stream (fread() then fclose()) or stream the whole file sequentially. No fseek() needed.

In such cases, it might be more efficient to do a fopen() on the stream directly, if possible. It seems that PHP allows fopen() on HTTP URLs. We could just stream the body of the response as it comes into the hooks and back to the client.

I had a quick look and it looks like most external storages could be modified to use streaming, as many of them use HTTP calls anyway.

The alternative to this would be to use a library like CURL that uses threads to pre-download the file into the temporary file. The control could be given back to the caller before the file is finished downloading, so that they can already start working on the start of the temporary file.

Please let me know what you think of this idea.
@icewind1991 @karlitschek @schiesbn @DeepDiver1975 @bantu

List of storages

• SMB (possible since 8.2 with the SMB library from @icewind1991 and libphp5-smbclient)
• FTP (might be possible with stream wrapper)
• SFTP
• WebDAV [WIP] Use stream wrapper that allows to seek HTTP files #11000 stream webdav downloads using http client #18653
• ownCloud
• Federated shares
• S3
• SWIFT: Streaming download from Swift in files_external #19002
• Dropbox
• GDrive: Stream upload/download for GDrive #21357

[PVince81]

Here is an example how to use streams with fopen():
http://php.net/manual/en/wrappers.http.php

[PVince81]

@schiesbn the only fseek() call I know so far is the one to detect whether a file is encrypted. As we discussed before, this can be replaced with an approach that simply detects whether keys exist for that file.

[karlitschek]

And additionally we could experiment with parallel http request to reduce roundtrip time. .
https://code.google.com/p/rolling-curl/
https://github.com/owncloud/administration/blob/master/performance-tests/oc-stress.php

[PVince81]

Just saw that the webdav.php storage already uses curl, but it still does a full-download of the files.
It also uses "php://temp" as temporary file instead of \OC_Helper::tmpFile($ext).
From the doc it seems that "php://temp" is using memory for up to 2 MB.

And the FTP storage does a fopen() directly on the ftp:// URL.
Seems to work with http as well (haven't tried https yet)

[PVince81]

• Check whether thumbnail creation is using streaming (direct fopen) or temporary files (@georgehrke)
• Find a way to stream with fopen() into avconv/ffmpeg instead of using a temp file (or raise an issue for that)

[PVince81]

For OC7 we will try and introduce streaming to WebDAV mounts and SMB mounts.
If time is left, then try and fix the other ones as well.

[bantu]

@PVince81 Doing WebDAV streaming should be not too hard to implement. How do you want to approach SMB/CIFS streaming?

[PVince81]

Piping with smbclient. I've already experimented with it and it works, see #4770 (comment)

But first we need to get @icewind1991 's new SMB impl in: #4770

[georgehrke]

Find a way to stream with fopen() into avconv/ffmpeg instead of using a temp file (or raise an issue for that)

We don't create a temp file of the whole file, just for the first 5MB.
https://github.com/owncloud/core/blob/master/lib/private/preview/movies.php#L47

[PVince81]

@georgehrke yes I know, that's why streaming ext storage will improve that as well.
Currently most ext storage will FIRST download the full file to the temp storage, then call fopen on the temp file and give the stream to you (the preview code)