ts-proto and prost inconsistencies when serializing empty numeric arrays

[haardikk21]

Hey,

I find myself in a situation trying to translate a TypeScript codebase to Rust, while maintaining compatibility and not breaking things.

On the TypeScript side of things, ts-proto has been used to generate bindings - and on the Rust side I've been using prost and tonic - which have been amazing so far btw thank you!

Recently, I realized that one specific field in one of our messages has inconsistencies in it's serialization format between the two implementations. Specifically, numeric arrays are treated differently when encoding to bytes.

In ts-proto, given a numeric array field on a message, if the array is empty - it's bytes serialization includes two zero bytes

In prost, given an empty numeric array, the field is skipped entirely and not part of the serialization at all

Unfortunately I cannot change the message structure.

What I'd like to ideally do is have a way to override the encode_raw function generated through prost_build for that message type - but that message is further present as a property in other messages and isn't a top-level object by itself.

Being new to Rust I also recently learnt it's not possible to override methods in trait implementations - and ideally I don't want to fork prost and maintain that myself

Now, I'm seeking help from the community here who have more experience with both Rust and prost than I do - here's the options I find myself with right now, but please suggest more if you have any:

1. Embed a minimal typescript function that uses ts-proto for embeddings inside Rust and call it using the JS Sandbox
2. Fork prost and maintain a slightly modified version which includes empty numeric arrays in the bytes serialization in the same way as ts-proto does

What would you suggest?

[LucioFranco]

If there are inconsistency one is likely wrong. If you have some sort of reproduction that would be helpful.

[haardikk21]

Sorry for the late reply. I figured it out.

If we have numeric optional arrays in the proto message, and:

if that field is constructed with null or undefined in ts-proto - that behaviour matches constructing it with None using prost

the problem is if that field is constructed using an empty array. in ts-proto it will still include the key and 2 empty bytes for that field, whereas prost will treat it the same way as None

I don't know which one matches the standard per-se, but for now I was able to get around this by creating a patch since I cannot change the ts-proto implementation. But it's not necessarily something wrong with prost.

[haardikk21]

my patched fork: https://github.com/OpenFarcaster/prost

[kriswuollett]

Both implementations may be correctly implemented with regards to standard unset/empty fields. The Protobuf programming guide mentions that:

The default value for repeated fields is empty (generally an empty list in the appropriate language).

Also check out the guides page on using field presence in generated code. It's up to the programming language and the deserialized message implementation on how to represent default, and the explicitly non-present values when using optional.

Apparently it looks like ts-proto implements proto3 optional. But on the other hand, prost currently does not implement optional, see #457. I do not have any suggestions on whether or not optional would be appropriate for you.