Stream upload/download for GDrive

[PVince81]

Currently the whole file is stored in memory and then passed to the library: https://github.com/owncloud/core/blob/v8.2.2/apps/files_external/lib/google.php#L469

This could easily cause out of memory errors in case the file is too big to fit in memory.

The question here is whether the library that OC uses called google-api-php-client supports stream upload ?

@icewind1991 @Xenopathic

[PVince81]

If direct stream upload (from php input to google) is not possible, at least it should stream from the TMP file and not first load it all in memory...

[PVince81]

https://developers.google.com/api-client-library/php/guide/media_upload

[PVince81]

Looking a the code of the library, it seems that even downloading a file will store it in memory first through curl_exec's response: https://github.com/owncloud/core/blob/v8.2.2/apps/files_external/3rdparty/google-api-php-client/src/Google/IO/Curl.php#L85

[PVince81]

Seems the library has a Stream class, maybe that could be used instead https://github.com/owncloud/core/blob/v8.2.2/apps/files_external/3rdparty/google-api-php-client/src/Google/IO/Stream.php ?

[PVince81]

Looks like it's possible to tell the client to use its stream class, by adding this:

            $config = new \Google_Config();
            $config->setIoClass("Google_IO_Stream");
            $this->client = new \Google_Client($config);

I see that this class uses stream_context_create but still requires the full content to be given.
A quick googling seems to tell that stream_context_create cannot be used directly with a file handle. (http://stackoverflow.com/a/1691616)

This means the full file would still be in memory...

And for download, even though the response could be a file handle, that class still copies it to memory: https://github.com/owncloud/core/blob/v8.2.2/apps/files_external/3rdparty/google-api-php-client/src/Google/IO/Stream.php#L128

It almost looks like the library wasn't designed with download/upload of big files in mind, but rather as a client for various APIs.

[PVince81]

If streaming is not possible, then at least "download directly to temp file" would still be better than "download to PHP string then save to temp file", at least would cause memory peak usage.

[PVince81]

If we anyway need a new library, maybe it's an opportunity to look into using Flysystem's GDrive implementation, if suitable ? (need to check that it doesn't store everything in memory...)

Just checked, and there is currently none 😦
It was requested here: thephpleague/flysystem#192

[PVince81]

Looks like someone started something here https://github.com/ignited/flysystem-google-drive (lots of TODOs) but they are using the same library so it will suffer from the same problems 😦