Support for streaming to a MySQL parameter from System.IO.Stream?

[kendallb]

Working on porting Rebus to MySqlConnector, and there is support for streaming from a source directly into the database via a database parameter, but this appears to not be supported with MySqlConnector? is this a limitation of MySqlConnector, or a limitation of the MySQL wire protocol itself? Streaming the result set works, I just cannot find a way to stream data on insert?

System.NotSupportedException : Parameter type MemoryStream (DbType: Binary) not currently supported. Value: System.IO.MemoryStream

I am using it like this (stolen directly from the SQL Server version):

command.Parameters.Add("data", MySqlDbType.VarBinary, MathUtil.GetNextPowerOfTwo((int)source.Length)).Value = source;

[kendallb]

I am getting it working by simply turning it into a byte array and submitting that, but that is not particularly ideal if the data is large.

[kendallb]

Code in question is here:

https://github.com/kendallb/Rebus.MySqlConnector/blob/master/Rebus.MySqlConnector/MySqlConnector/DataBus/MySqlDataBusStorage.cs#L113

[bgrainger]

is this a limitation of MySqlConnector, or a limitation of the MySQL wire protocol itself?

A little bit of both. 😀

MySqlConnector doesn't have support for it, hence the exception.

It could certainly be added (as a convenience method for this use case). However, the MySQL text protocol requires the entire INSERT SQL statement to be sent as a length-prefixed packet. Thus, MySqlConnector could have to convert the entire Stream to an in-memory blob; it couldn't really be streamed. (Various reserved bytes in the Stream have to be escaped, so the length can't be known ahead of time just by checking Stream.Length. MySqlConnector could read the Stream twice, once to compute the length, and once to send it, but this might be inefficient or just not even supported for some forward-only Streams.)

Still, this is probably worth adding, even if just for API compatibility with other ADO.NET providers, even if it doesn't support actual streaming, at least initially.

[kendallb]

Right, it would be nice to support it just to make it more compatible with other ADO.net connectors like SQL Server, even if under the hood it just does what I did, and convert the stream into a byte array. That does mean it would crash out if someone tries to insert a massive blob, but most folks don't have the max packet size for MySQL set to a massive number anyway (in our case it's 16MB for production) so unless the MySQL wire protocol supported proper streaming, it would still be limited to whatever that size is. Clearly SQL Server supports massive blobs as the Rebus unit test for the code in question inserts a 100MB file, which naturally crapped itself on my test system with a 16MB max packet size :)

At the end of the day folks shouldn't be inserting massive blobs into MySQL anyway, as it's the wrong tool for the job. if you need to store massive blobs, they should be stored in a storage bucket somewhere (Amazon S3, Google etc) and a link to the resource stored in MySQL, so I don't think it's a big limitation.

But, it would be nice when porting code like this for it to just 'work', at least until you actually try to insert something massive :).

[bgrainger]

That does mean it would crash out if someone tries to insert a massive blob, but most folks don't have the max packet size for MySQL set to a massive number anyway (in our case it's 16MB for production)

That's a good point: usually one is quite limited in the maximum blob size they can insert, due to this limit.

[bgrainger]

Updated ADO.NET tests; it appears that SqlClient is the only other library that supports using a Stream as a parameter value.

Npgsql has decided not to support it (and throws a NotSupportedException): npgsql/npgsql#550

Most other libraries "succeed" but write "System.IO.MemoryStream" to the database column, introducing silent data corruption.

[kendallb]

Yeah it's 6 of one, half a dozen of another. When you get a nice exception at least you know stuff is not working and can fix it. In some ways I suppose having the developer specifically write the code like I did to turn it into a byte array means you are then fully aware of the consequences (memory usage). That might be better than having someone think they can stream a 1G file into a database blob only to find it runs out of memory :)

[bgrainger]

I'm adding support for MemoryStream. If TryGetBuffer succeeds, that will be written directly to the packet (which should be efficient). Otherwise, ToArray() will be called, which will duplicate all the data before copying it (inefficient).

A different approach (e.g., that read repeatedly into a 64KiB buffer) could be implemented, which would work for all Stream types. That could replace the ToArray() code path in the future if that were a pressing need.

(A concern, as discussed above, with supporting any arbitrary Stream is that it might make people think they can stream a very large binary file to MySQL Server, but that's not actually possible.)

[kendallb]

Ok, I think supporting any stream should be implemented to make it really useful. A memory stream itself is not all that useful as all the data will already be in memory in that case so not sure it buys you much other than avoiding the second copy. A lot of the time it’s likely to be a real stream like a file stream or network stream, so the buffering approach would make the most sense to me?

If the buffering approach is implemented, does that mean it would avoid the max packet size issue? Or will that still be a problem? If it can avoid that then at least it would be possible to stream large blob data in, larger that the max packer size.