My experience with proto generation

[antspy]

Hi,

Just wanted to share my experience with proto generation and bazel. I have done quite a bit of work to ensure that it worked correctly on my machine, and it unfortunately involved a good amount of hacks.

I am not claiming this is a perfect setup - I am not an expert on bazel, so there might be better solutions that I'm ignoring. But this is a review of the challenges I found and the way I solved them. I will be happy to provide more pointers for some parts if needed.

If this ends up in a more streamlined usage of protos, that would be great!

Supporting proto

Supporting protos mean a few things:

1. Have a proto_library that bazel can build, so your code compiles. This works nicely, as all the macros are already there.
2. Integrate it with the editor, so that it knows what types are present and how are the protos structured.
3. Making sure that aspect configure understand the proto dependencies, both in .proto files and in typescript / python files.

In order to support 2, it's virtually unavoidable to copy the generated files back to the source tree (I guess the other option is to integrate with the editor and tell the editor where it should look for the generated files, but this seems hard. I have not tried it, however).

Also, in order to avoid cluttering the source tree, I want to put the generated files in a separate directory.

src/
   example.proto
    generated_proto/
      example/
        BUILD
        example_pb2.py
        example_pb2.pyi
        ...

Let me now list some of the issues I found:

py_proto_library does not generate pyi files.

The example_pb2.py file is useful at runtime, but it's useless for the editor to understand the type of the protos. Given that py_proto_library does not generate pyi files, I needed to call protoc myself and copy the file in the right directory.

ts_proto_library.copy does not allow changing directory structure

While the ts_proto_library has useful .copy targets that can be used to copy the generated files back to the source tree, they also include a test to ensure that those generated files are in sync with the ones in the bazel-bin directory. While this is good in theory, it becomes problematic if I want the files to live in the generated_proto directory.

I had to set copy_files=False in the ts_proto_library and build and copy the files myself in the right directory.

This manual generation requires custom diff test

As mentioned above, we cannot rely on bazel generation by itself at the moemnt. We need to have a bunch of files created using protoc, then manually copied from the bazel-bin dir to the source tree. So we also need to add some logic that the generated files are not out of sync with the .proto file.

I have already a setup for this (used elsewhere in my codebease), so it was not a problem, but it's something to keep in mind.

[alexeagle]

Again it would be easier if you could file separate issues for separate things, since they have different metadata (ie. "closed")

py_proto_library does not generate pyi files.

Since you found a solution for that on Bazel slack, could you send a PR for that one?

[antspy]

There's a PR being reviewed here: protocolbuffers/protobuf#21567

Although it's taking some time :)

[alexeagle]

Neat! My understanding is there's nothing more we can do in toolchains_protoc so I'll close this issue, tell me if I'm wrong please :)