This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ozone.git
The following commit(s) were added to refs/heads/master by this push:
new 9e4da5bac2 HDDS-12821. Update Build from Source user doc. (#8262)
9e4da5bac2 is described below
commit 9e4da5bac25a356ff3704cc7d5f471757fbf08b2
Author: Wei-Chiu Chuang <[email protected]>
AuthorDate: Mon Apr 14 11:15:15 2025 -0700
HDDS-12821. Update Build from Source user doc. (#8262)
---
hadoop-hdds/docs/content/start/FromSource.md | 91 ++++++++++++++++++++++++----
1 file changed, 80 insertions(+), 11 deletions(-)
diff --git a/hadoop-hdds/docs/content/start/FromSource.md
b/hadoop-hdds/docs/content/start/FromSource.md
index fcbc897d37..9d3947ad1c 100644
--- a/hadoop-hdds/docs/content/start/FromSource.md
+++ b/hadoop-hdds/docs/content/start/FromSource.md
@@ -20,8 +20,15 @@ weight: 30
-->
{{< requirements >}}
- * Java 1.8
- * Maven
+* x86 or ARM64
+* Linux or MacOS
+* JDK 1.8 or higher
+* [Maven 3.6 or later](https://maven.apache.org/download.cgi)
+* Internet connection for first build (to fetch all Maven and Ozone
dependencies)
+
+You will also need to install the following dependencies to build native code
(optional):
+* gcc
+* cmake
{{< /requirements >}}
<div class="alert alert-info" role="alert">
@@ -32,29 +39,91 @@ planning to build sources yourself, you can safely skip
this page.
</div>
-If you are a Hadoop ninja, and wise in the ways of Apache, you already know
-that a real Apache release is a source release.
-
If you want to build from sources, Please untar the source tarball (or clone
the latest code
-from the [git repository](https://github.com/apache/ozone)) and run the ozone
build command. This instruction assumes that you have all the
-dependencies to build Hadoop on your build machine. If you need instructions
-on how to build Hadoop, please look at the Apache Hadoop Website.
+from the [git repository](https://github.com/apache/ozone)).
+
+## ARM-based Linux
+If you are using an ARM-based Linux, patch protobuf 2.5.0 and build it from
source.
```bash
-mvn clean package -DskipTests=true
+sudo yum install -y make cmake gcc g++ patch
+# or for Ubuntu and Debian:
+# sudo apt-get install -y make cmake gcc g++ patch
+
+# Download protobuf 2.5.0 tarball
+curl -sSL
https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
| tar zx
+cd protobuf-2.5.0
+
+# patch protobuf 2.5.0
+curl -L -O
https://gist.githubusercontent.com/liusheng/64aee1b27de037f8b9ccf1873b82c413/raw/118c2fce733a9a62a03281753572a45b6efb8639/protobuf-2.5.0-arm64.patch
+patch -p1 < protobuf-2.5.0-arm64.patch
+# build protobuf
+./configure --disable-shared
+make
+# install protoc to the local Maven repository
+mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc
-Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=src/protoc
+# workaround for Maven 3.9.x
+cp
$HOME/.m2/repository/com/google/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64
$HOME/.m2/repository/com/google/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64.exe
+```
+
+## ARM-based Apple Silicon (Apple M1 ... etc)
+
+```bash
+PROTOBUF_VERSION="3.7.1"
+curl -sSL
https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-all-${PROTOBUF_VERSION}.tar.gz
| tar zx
+cd protobuf-${PROTOBUF_VERSION}
+./configure --disable-shared
+make -j
+# install protoc to the local Maven repository
+mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc
-Dversion=${PROTOBUF_VERSION} -Dclassifier=osx-aarch_64 -Dpackaging=exe
-Dfile=src/protoc
+# workaround for Maven 3.9.x. Not needed for 3.8.x or earlier
+cp
$HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64
$HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64.exe
+
+cd ..
+# Download protobuf 2.5.0 tarball
+PROTOBUF_VERSION="2.5.0"
+curl -sSL
https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-${PROTOBUF_VERSION}.tar.gz
| tar zx
+cd protobuf-${PROTOBUF_VERSION}
+
+# patch protobuf 2.5.0
+curl -L -O
https://gist.githubusercontent.com/liusheng/64aee1b27de037f8b9ccf1873b82c413/raw/118c2fce733a9a62a03281753572a45b6efb8639/protobuf-2.5.0-arm64.patch
+patch -p1 < protobuf-2.5.0-arm64.patch
+# build protobuf
+./configure --disable-shared
+make
+# install protoc to the local Maven repository
+mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc
-Dversion=${PROTOBUF_VERSION} -Dclassifier=osx-aarch_64 -Dpackaging=exe
-Dfile=src/protoc
+# workaround for Maven 3.9.x. Not needed for 3.8.x or earlier
+cp
$HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64
$HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64.exe
```
+## Build Ozone
+Run the ozone build command.
+
+```bash
+mvn clean package -DskipTests=true
+```
This will build an `ozone-\<version\>` directory in your
`hadoop-ozone/dist/target` directory.
You can copy this tarball and use this instead of binary artifacts that are
provided along with the official release.
-To create tar file distribution, use the `-Pdist` profile:
+Depending on the network speed, the build can take anywhere from 10 minutes to
20 minutes.
+
+To create tarball file distribution, use the `-Pdist` profile:
```bash
mvn clean package -DskipTests=true -Pdist
```
+## Other useful Maven build options
+
+* Use `-DskipShade` to skip shaded Ozone FS jar file creation. Saves time, but
you can't test integration with other software that uses Ozone as a
Hadoop-compatible file system.
+* Use `-DskipRecon` to skip building Recon Web UI. It saves about 2 minutes.
+* Use `-Dmaven.javadoc.skip=true` to skip building javadocs.
+* Use `-Drocks_tools_native` to build the RocksDB native code for the Ozone
Snapshot feature. This is optional and not required for building Ozone. It is
only needed if you want to build the RocksDB native code for Ozone.
+
+
## How to run Ozone from build
When you have the new distribution, you can start a local cluster [with
docker-compose]({{< ref "start/RunningViaDocker.md">}}).
@@ -67,4 +136,4 @@ docker-compose up -d
## How to test the build
-`compose` subfolder contains multiple type of example setup (secure,
non-secure, HA, Yarn). They can be tested with the help of
[robotframework](http://robotframework.org/) with executing `test.sh` in any of
the directories.
\ No newline at end of file
+`compose` subfolder contains multiple type of example setup (secure,
non-secure, HA, Yarn). They can be tested with the help of
[robotframework](http://robotframework.org/) with executing `test.sh` in any of
the directories.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
