Skip to content

Commit 9e4da5b

Browse files
jojochuangjojochuang
authored
HDDS-12821. Update Build from Source user doc. (apache#8262)
1 parent 5292ac5 commit 9e4da5b

1 file changed

Lines changed: 80 additions & 11 deletions

File tree

‎hadoop-hdds/docs/content/start/FromSource.md‎

Lines changed: 80 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,15 @@ weight: 30
2020
-->
2121

2222
{{< requirements >}}
23-
* Java 1.8
24-
* Maven
23+
* x86 or ARM64
24+
* Linux or MacOS
25+
* JDK 1.8 or higher
26+
* [Maven 3.6 or later](https://maven.apache.org/download.cgi)
27+
* Internet connection for first build (to fetch all Maven and Ozone dependencies)
28+
29+
You will also need to install the following dependencies to build native code (optional):
30+
* gcc
31+
* cmake
2532
{{< /requirements >}}
2633

2734
<div class="alert alert-info" role="alert">
@@ -32,29 +39,91 @@ planning to build sources yourself, you can safely skip this page.
3239

3340
</div>
3441

35-
If you are a Hadoop ninja, and wise in the ways of Apache, you already know
36-
that a real Apache release is a source release.
37-
3842
If you want to build from sources, Please untar the source tarball (or clone the latest code
39-
from the [git repository](https://github.com/apache/ozone)) and run the ozone build command. This instruction assumes that you have all the
40-
dependencies to build Hadoop on your build machine. If you need instructions
41-
on how to build Hadoop, please look at the Apache Hadoop Website.
43+
from the [git repository](https://github.com/apache/ozone)).
44+
45+
## ARM-based Linux
46+
If you are using an ARM-based Linux, patch protobuf 2.5.0 and build it from source.
4247

4348
```bash
44-
mvn clean package -DskipTests=true
49+
sudo yum install -y make cmake gcc g++ patch
50+
# or for Ubuntu and Debian:
51+
# sudo apt-get install -y make cmake gcc g++ patch
52+
53+
# Download protobuf 2.5.0 tarball
54+
curl -sSL https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz | tar zx
55+
cd protobuf-2.5.0
56+
57+
# patch protobuf 2.5.0
58+
curl -L -O https://gist.githubusercontent.com/liusheng/64aee1b27de037f8b9ccf1873b82c413/raw/118c2fce733a9a62a03281753572a45b6efb8639/protobuf-2.5.0-arm64.patch
59+
patch -p1 < protobuf-2.5.0-arm64.patch
60+
# build protobuf
61+
./configure --disable-shared
62+
make
63+
# install protoc to the local Maven repository
64+
mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=src/protoc
65+
# workaround for Maven 3.9.x
66+
cp $HOME/.m2/repository/com/google/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64 $HOME/.m2/repository/com/google/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64.exe
67+
```
68+
69+
## ARM-based Apple Silicon (Apple M1 ... etc)
70+
71+
```bash
72+
PROTOBUF_VERSION="3.7.1"
73+
curl -sSL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-all-${PROTOBUF_VERSION}.tar.gz | tar zx
74+
cd protobuf-${PROTOBUF_VERSION}
75+
./configure --disable-shared
76+
make -j
77+
# install protoc to the local Maven repository
78+
mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=${PROTOBUF_VERSION} -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=src/protoc
79+
# workaround for Maven 3.9.x. Not needed for 3.8.x or earlier
80+
cp $HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64 $HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64.exe
81+
82+
cd ..
83+
# Download protobuf 2.5.0 tarball
84+
PROTOBUF_VERSION="2.5.0"
85+
curl -sSL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-${PROTOBUF_VERSION}.tar.gz | tar zx
86+
cd protobuf-${PROTOBUF_VERSION}
87+
88+
# patch protobuf 2.5.0
89+
curl -L -O https://gist.githubusercontent.com/liusheng/64aee1b27de037f8b9ccf1873b82c413/raw/118c2fce733a9a62a03281753572a45b6efb8639/protobuf-2.5.0-arm64.patch
90+
patch -p1 < protobuf-2.5.0-arm64.patch
91+
# build protobuf
92+
./configure --disable-shared
93+
make
94+
# install protoc to the local Maven repository
95+
mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=${PROTOBUF_VERSION} -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=src/protoc
96+
# workaround for Maven 3.9.x. Not needed for 3.8.x or earlier
97+
cp $HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64 $HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64.exe
4598
```
4699

100+
## Build Ozone
101+
Run the ozone build command.
102+
103+
```bash
104+
mvn clean package -DskipTests=true
105+
```
47106
This will build an `ozone-\<version\>` directory in your `hadoop-ozone/dist/target` directory.
48107

49108
You can copy this tarball and use this instead of binary artifacts that are
50109
provided along with the official release.
51110

52-
To create tar file distribution, use the `-Pdist` profile:
111+
Depending on the network speed, the build can take anywhere from 10 minutes to 20 minutes.
112+
113+
To create tarball file distribution, use the `-Pdist` profile:
53114

54115
```bash
55116
mvn clean package -DskipTests=true -Pdist
56117
```
57118

119+
## Other useful Maven build options
120+
121+
* Use `-DskipShade` to skip shaded Ozone FS jar file creation. Saves time, but you can't test integration with other software that uses Ozone as a Hadoop-compatible file system.
122+
* Use `-DskipRecon` to skip building Recon Web UI. It saves about 2 minutes.
123+
* Use `-Dmaven.javadoc.skip=true` to skip building javadocs.
124+
* Use `-Drocks_tools_native` to build the RocksDB native code for the Ozone Snapshot feature. This is optional and not required for building Ozone. It is only needed if you want to build the RocksDB native code for Ozone.
125+
126+
58127
## How to run Ozone from build
59128

60129
When you have the new distribution, you can start a local cluster [with docker-compose]({{< ref "start/RunningViaDocker.md">}}).
@@ -67,4 +136,4 @@ docker-compose up -d
67136

68137
## How to test the build
69138

70-
`compose` subfolder contains multiple type of example setup (secure, non-secure, HA, Yarn). They can be tested with the help of [robotframework](http://robotframework.org/) with executing `test.sh` in any of the directories.
139+
`compose` subfolder contains multiple type of example setup (secure, non-secure, HA, Yarn). They can be tested with the help of [robotframework](http://robotframework.org/) with executing `test.sh` in any of the directories.

0 commit comments

Comments
 (0)