github-actions[bot] commented on code in PR #67330:
URL: https://github.com/apache/doris/pull/67330#discussion_r3900344839
##########
build.sh:
##########
@@ -503,27 +491,12 @@ fi
# shellcheck source=thirdparty/arrow-paimon-vars.sh
. "${DORIS_HOME}/thirdparty/arrow-paimon-vars.sh"
NEED_ARROW_PAIMON_THIRDPARTY=false
-if [[ "${BUILD_BE}" -eq 1 || "${BUILD_META_TOOL}" == "ON" ||
- "${BUILD_FILE_CACHE_MICROBENCH_TOOL}" == "ON" ||
+if [[ "${BUILD_BE}" -eq 1 || "${BUILD_CLOUD}" -eq 1 ||
+ "${BUILD_META_TOOL}" == "ON" || "${BUILD_FILE_CACHE_MICROBENCH_TOOL}" ==
"ON" ||
Review Comment:
[P1] Do not require the BE Arrow/Paimon stack for Cloud-only builds
`cloud/cmake/thirdparty.cmake` imports neither Arrow nor Paimon, but this
condition makes `./build.sh --cloud` validate them anyway. With an
external/install-only prefix whose BE stack is stale or absent (including the
current layout transition), the build enters `rebuild_thirdparty_libraries()`
and exits because no source builder exists before Cloud configuration starts.
Remove `BUILD_CLOUD` from this BE-stack gate and restore the Cloud-only
dispatch case.
##########
be/CMakeLists.txt:
##########
@@ -200,6 +200,7 @@ option(BUILD_FILE_CACHE_LRU_TOOL "ON for building file
cache lru tool or OFF for
message(STATUS "build file cache lru tool: ${BUILD_FILE_CACHE_LRU_TOOL}")
option(ENABLE_PAIMON_CPP "Enable Paimon C++ integration" ON)
+set(PAIMON_HOME "" CACHE PATH "Paimon install prefix")
option(ENABLE_TDE "Enable TDE feature module" OFF)
Review Comment:
[P2] Do not accept an ignored Paimon install prefix
This cache/env input no longer selects anything: the PR removes
`${PAIMON_HOME}/include`, every Paimon archive in `be/cmake/thirdparty.cmake`
now comes from `${THIRDPARTY_DIR}`, and no other `PAIMON_HOME` reference
remains. A caller can therefore set a valid custom prefix and silently link the
default tree. Either honor the same root consistently for preflight, headers,
and libraries, or remove/reject the unsupported override explicitly.
##########
build.sh:
##########
@@ -757,6 +728,16 @@ for ((i = 0; i < ${#CLOUD_EXTRA_FEATURE_KEYS[@]}; i++)); do
done
if [[ "${COMPILE_BENCH}" -eq 1 ]]; then
+ # BE compile benchmark mode: measure a cold, cache-free BE C++ build.
+ # Everything that is not the BE C++ build would only add noise, so force
+ # a BE-only build regardless of the other options.
+ BUILD_BE=1
+ BUILD_FE=0
Review Comment:
[P2] Normalize compile-bench before the clean-only exit
Standalone `--compile-bench` leaves every `BUILD_*` target zero until this
block. With `--clean`, the earlier clean-only branch therefore exits
successfully without a benchmark; without it, Arrow/Paimon preflight is skipped
even though the mode later configures BE (so a stale or dual-layout root is
used unvalidated). Move this BE-target normalization before both gates and
restore dispatch coverage for standalone and `--clean` benchmarks.
##########
thirdparty/build-thirdparty.sh:
##########
@@ -1109,19 +1103,11 @@ build_grpc() {
# sed -i 's/find_dependency/find_package/g'
"${TP_INSTALL_DIR}"/lib64/cmake/grpc/gRPCConfig.cmake
}
-# Arrow 17 is installed in the legacy unversioned prefix for pre-upgrade
-# branch-4.1 revisions, while Arrow 24 is installed in a versioned prefix
-# selected by master.
-build_arrow_stack() {
- local arrow_source="$1"
- local xsimd_archive="$2"
- local install_dir="$3"
- local has_separate_compute_archive="$4"
-
- check_if_source_exist "${arrow_source}"
- mkdir -p "${install_dir}/lib64"
- ln -sfn lib64 "${install_dir}/lib"
- cd "${TP_SOURCE_DIR}/${arrow_source}/cpp"
+# arrow
+build_arrow() {
+ check_if_source_exist "${ARROW_SOURCE}"
+ invalidate_arrow_prebuilt_marker "${TP_INSTALL_DIR}"
+ cd "${TP_SOURCE_DIR}/${ARROW_SOURCE}/cpp"
Review Comment:
[P1] Invalidate the paired Paimon install before overwriting root Arrow
A dual-layout tree still has Arrow/Paimon 17 in the root. This recovery
invalidates only Arrow, installs and publishes root Arrow 24, and does not
invalidate or replace root Paimon 17 until `build_paimon_cpp()` starts. If the
process is interrupted in that window, `run-be-ut.sh` can configure BE directly
from the mixed root without calling the pair validator, with Paimon enabled by
default; the imported Paimon 17 archives expose Arrow types and are not
ABI-safe with Arrow 24. Invalidate/remove the co-located Paimon artifacts
before overwriting Arrow (or stage/swap the pair atomically), and add a
dual-to-single interruption case covering the direct BE entry point.
##########
build.sh:
##########
@@ -562,12 +535,10 @@ rebuild_thirdparty_libraries() {
if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ]]; then
echo "Thirdparty libraries need to be build ..."
rebuild_thirdparty_libraries true
-elif [[ "${NEED_ARROW_PAIMON_THIRDPARTY}" == "true" ]]; then
- select_arrow_paimon_rebuild_packages "${DORIS_THIRDPARTY}/installed"
- if [[ "${#ARROW_PAIMON_REBUILD_PACKAGES[@]}" -gt 0 ]]; then
- echo "Arrow/Paimon thirdparty libraries need to be rebuilt ..."
- rebuild_thirdparty_libraries false
"${ARROW_PAIMON_REBUILD_PACKAGES[@]}"
- fi
+elif [[ "${NEED_ARROW_PAIMON_THIRDPARTY}" == "true" ]] &&
+ ! arrow_paimon_prebuilt_valid "${DORIS_THIRDPARTY}/installed"; then
+ echo "Arrow/Paimon thirdparty libraries need to be rebuilt ..."
+ rebuild_thirdparty_libraries false "${ARROW_PAIMON_BUILD_PACKAGES[@]}"
fi
Review Comment:
[P1] Make the shared prebuilt acceptable to live branch-4.1
This rebuild will publish the root Paimon marker
`9887cf1ec13106da4250f8d7ea931a258879089c`, but live `branch-4.1`
(`211f6165abaf`) validates the same `automation` asset against
`cb82e41ba46f534e611cdd52e66b53c227d49bf8` and only permits legacy marker
`dbb6ca6e243cb3aa783b7a8011f97afda9e7ea28`. Its staged archive installer
therefore rejects a prebuilt produced by this revert before replacing
`installed`; the branch-4.1 dual-layout companion (#67320) was closed.
Coordinate a branch-4.1 compatibility update (or preserve a marker it accepts)
and cover that live validator contract before publishing the reverted asset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
