[
https://issues.apache.org/jira/browse/HADOOP-19663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18016942#comment-18016942
]
ASF GitHub Bot commented on HADOOP-19663:
-----------------------------------------
PeterPtroc commented on PR #7903:
URL: https://github.com/apache/hadoop/pull/7903#issuecomment-3235836592
@pan3793 Thanks for the suggestion! Below is a concise doc to verify the
correctness of the crc32riscv implementation:
I mainly verify on RISC‑V by using QEMU together with the openEuler RISC‑V
image.
Download the image
-
https://dl-cdn.openeuler.openatom.cn/openEuler-25.03/virtual_machine_img/riscv64/
For me, from the above link, download these four files: RISCV_VIRT_CODE.fd,
RISCV_VIRT_VARS.fd, openEuler-25.03-riscv64.qcow2.xz, and start_vm.sh; then log
in as root with the password: openEuler12#$.
Install required packages
````bash
yum install -y gcc gcc-c++ gcc-gfortran libgcc cmake
yum install -y wget openssl openssl-devel zlib zlib-devel automake libtool
make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel
doxygen clang cyrus-sasl cyrus-sasl-devel libtirpc libtirpc-devel
yum install -y java-17-openjdk.riscv64 java-17-openjdk-devel.riscv64
java-17-openjdk-headless.riscv64
````
Install Protobuf 2.5.0 (with RISC‑V patches)
````bash
mkdir protobuf && cd protobuf
# Fetch sources
git clone https://gitee.com/src-openeuler/protobuf2.git
cd protobuf2
tar -xjf protobuf-2.5.0.tar.bz2
cp *.patch protobuf-2.5.0 && cd protobuf-2.5.0
# Apply patches (adds riscv64 support and build fixes)
patch -p1 < 0001-Add-generic-GCC-support-for-atomic-operations.patch
patch -p1 < protobuf-2.5.0-gtest.patch
patch -p1 < protobuf-2.5.0-java-fixes.patch
patch -p1 < protobuf-2.5.0-makefile.patch
patch -p1 < add-riscv64-support.patch
# Autotools setup
libtoolize
yum install -y automake
automake-1.17 -a
chmod +x configure
# Configure, build, install
./configure --build=riscv64-unknown-linux --prefix=/usr/local/protobuf-2.5.0
make
make check
make install
ldconfig
# Publish protoc 2.5.0 into local Maven repo (riscv64 classifier)
mvn install:install-file \
-DgroupId=com.google.protobuf \
-DartifactId=protoc \
-Dversion=2.5.0 \
-Dclassifier=linux-riscv64 \
-Dpackaging=exe \
-Dfile=/usr/local/protobuf-2.5.0/bin/protoc
cd ..
````
Install Protobuf 3.25.5
````bash
# Download and unpack
wget -c
https://github.com/protocolbuffers/protobuf/releases/download/v25.5/protobuf-25.5.tar.gz
tar -xzf protobuf-25.5.tar.gz
cd protobuf-25.5
# Abseil dependency
git clone https://github.com/abseil/abseil-cpp third_party/abseil-cpp
# Configure and build
cmake ./ \
-DCMAKE_BUILD_TYPE=RELEASE \
-Dprotobuf_BUILD_TESTS=off \
-DCMAKE_CXX_STANDARD=20 \
-DCMAKE_INSTALL_PREFIX=/usr/local/protobuf-3.25.5
make install -j "$(nproc)"
# Publish protoc 3.25.5 into local Maven repo (riscv64 classifier)
mvn install:install-file \
-DgroupId=com.google.protobuf \
-DartifactId=protoc \
-Dversion=3.25.5 \
-Dclassifier=linux-riscv64 \
-Dpackaging=exe \
-Dfile=/usr/local/protobuf-3.25.5/bin/protoc
# Make protoc available on PATH and verify
sudo ln -sfn /usr/local/protobuf-3.25.5/bin/protoc /usr/local/bin/protoc
protoc --version
````
Verify CRC32 using Hadoop native
````bash
# Clone Hadoop
git clone https://github.com/apache/hadoop.git
cd hadoop
# Increase Maven memory
export MAVEN_OPTS="-Xmx8g -Xms6g"
# Build Hadoop Common (native enabled)
nohup mvn -pl hadoop-common-project/hadoop-common -am -Pnative -DskipTests
clean install > build.log 2>&1 &
# Point to built native library directory
export
HADOOP_COMMON_LIB_NATIVE_DIR="$PWD/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib"
export LD_LIBRARY_PATH="$HADOOP_COMMON_LIB_NATIVE_DIR:$LD_LIBRARY_PATH"
# Run the CRC32 native test
nohup mvn -Pnative -Dtest=org.apache.hadoop.util.TestNativeCrc32 \
-Djava.library.path="$HADOOP_COMMON_LIB_NATIVE_DIR" test > test.log 2>&1 &
> Add RISC-V build infrastructure and placeholder implementation for CRC32
> acceleration
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-19663
> URL: https://issues.apache.org/jira/browse/HADOOP-19663
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
> Affects Versions: 3.5.0
> Reporter: Ptroc
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> Establish the foundational build infrastructure for RISC-V CRC32 hardware
> acceleration support while maintaining full backward compatibility with
> existing software implementations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
