Skip to content

[SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf. #38743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
WolverineJiang wants to merge 2 commits into from
Closed

[SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc executables when building Spark Protobuf. #38743

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
48f1483
[SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc execu…
Nov 21, 2022
5be2fe0
[SPARK-41215][BUILD][PROTOBUF] Support user configurable protoc execu…
Nov 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions connector/protobuf/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Spark Protobuf - Developer Documentation

## Getting Started

### Build

```bash
./build/mvn -Phive clean package
Copy link
Contributor

@LuciferYang LuciferYang Nov 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For protobuf module, do we have to compile with -Phive?

Copy link
Author

@WolverineJiang WolverineJiang Nov 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't seem to be necessary. Let me adjust it.

```

or

```bash
./build/sbt -Phive clean package
```

### Build with user-defined `protoc`

When the user cannot use the official `protoc` binary files to build the `protobuf` module in the compilation environment,
for example, compiling `protobuf` module on CentOS 6 or CentOS 7 which the default `glibc` version is less than 2.14, we can try to compile and test by
specifying the user-defined `protoc` binary files as follows:

```bash
export PROTOBUF_PROTOC_EXEC_PATH=/path-to-protoc-exe
./build/mvn -Phive -Puser-defined-protoc clean package
```

or

```bash
export PROTOBUF_PROTOC_EXEC_PATH=/path-to-protoc-exe
export CONNECT_PLUGIN_EXEC_PATH=/path-to-protoc-gen-grpc-java-exe
./build/sbt -Puser-defined-protoc clean package
```

The user-defined `protoc` binary files can be produced in the user's compilation environment by source code compilation,
for compilation steps, please refer to [protobuf](https://github.com/protocolbuffers/protobuf).
89 changes: 67 additions & 22 deletions connector/protobuf/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,28 +110,73 @@
</relocations>
</configuration>
</plugin>
<plugin>
<groupId>com.github.os72</groupId>
<artifactId>protoc-jar-maven-plugin</artifactId>
<version>3.11.4</version>
<!-- Generates Java classes for tests. TODO(Raghu): Generate descriptor files too. -->
<executions>
<execution>
<phase>generate-test-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<protocArtifact>com.google.protobuf:protoc:${protobuf.version}</protocArtifact>
<protocVersion>${protobuf.version}</protocVersion>
<inputDirectories>
<include>src/test/resources/protobuf</include>
</inputDirectories>
<addSources>test</addSources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>default-protoc</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<build>
<plugins>
<plugin>
<groupId>com.github.os72</groupId>
<artifactId>protoc-jar-maven-plugin</artifactId>
<version>3.11.4</version>
<!-- Generates Java classes for tests. TODO(Raghu): Generate descriptor files too. -->
<executions>
<execution>
<phase>generate-test-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<protocArtifact>com.google.protobuf:protoc:${protobuf.version}</protocArtifact>
<protocVersion>${protobuf.version}</protocVersion>
<inputDirectories>
<include>src/test/resources/protobuf</include>
</inputDirectories>
<addSources>test</addSources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>user-defined-protoc</id>
<properties>
<protobuf.protoc.executable.path>${env.PROTOBUF_PROTOC_EXEC_PATH}</protobuf.protoc.executable.path>
</properties>
<build>
<plugins>
<plugin>
<groupId>com.github.os72</groupId>
<artifactId>protoc-jar-maven-plugin</artifactId>
<version>3.11.4</version>
<!-- Generates Java classes for tests. TODO(Raghu): Generate descriptor files too. -->
<executions>
<execution>
<phase>generate-test-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<protocArtifact>com.google.protobuf:protoc:${protobuf.version}</protocArtifact>
<protocVersion>${protobuf.version}</protocVersion>
<protocCommand>${protobuf.protoc.executable.path}</protocCommand>
<inputDirectories>
<include>src/test/resources/protobuf</include>
</inputDirectories>
<addSources>test</addSources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
15 changes: 14 additions & 1 deletion project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,10 +112,14 @@ object SparkBuild extends PomBuild {
if (profiles.contains("user-defined-protoc")) {
val connectProtocExecPath = Properties.envOrNone("CONNECT_PROTOC_EXEC_PATH")
val connectPluginExecPath = Properties.envOrNone("CONNECT_PLUGIN_EXEC_PATH")
val protobufProtocExecPath = Properties.envOrNone("PROTOBUF_PROTOC_EXEC_PATH")
if (connectProtocExecPath.isDefined && connectPluginExecPath.isDefined) {
sys.props.put("connect.protoc.executable.path", connectProtocExecPath.get)
sys.props.put("connect.plugin.executable.path", connectPluginExecPath.get)
}
if (protobufProtocExecPath.isDefined) {
sys.props.put("protobuf.protoc.executable.path", protobufProtocExecPath.get)
}
}
profiles
}
Expand Down Expand Up @@ -779,7 +783,16 @@ object SparkProtobuf {
case m if m.toLowerCase(Locale.ROOT).endsWith(".proto") => MergeStrategy.discard
case _ => MergeStrategy.first
},
)
) ++ {
val protobufProtocExecPath = sys.props.get("protobuf.protoc.executable.path")
if (protobufProtocExecPath.isDefined) {
Seq(
PB.protocExecutable := file(protobufProtocExecPath.get)
)
} else {
Seq.empty
}
}
}

object Unsafe {
Expand Down