jonkeane opened a new issue, #55478:
URL: https://github.com/apache/arrow/issues/55478
### Describe the enhancement requested
When the R package builds libarrow from source without network access (e.g.
on CRAN), `nixlibs.R` calls `turn_off_all_optional_features()`. That build has
no Parquet, Dataset, JSON, compression codecs, RE2 or utf8proc. #55461
(GH-55454) added a way to ship thirdparty source tarballs inside the R package
(`tools/vendored`, populated by `tools/vendor_dependencies.sh` during `make
sync-cpp`) for xsimd and uriparser, which even a minimal build needs. The same
mechanism could ship everything a full non-cloud build needs, including
Parquet, and keep the package small.
**Summary:** trimmed tarballs for every non-cloud dependency (Parquet,
Dataset, Acero, JSON, all codecs, RE2, utf8proc, mimalloc) add **~9.9 MB** to
the package (~14 MB total, compared with 4.2 MB today). Untrimmed they would
add ~157 MB, almost all of it Boost (136 MB). I built libarrow from the trimmed
tarballs with no downloads on macOS arm64 (details below).
### Current size
- arrow 26.0.1 on CRAN: **4.2 MB**
- \+ xsimd and uriparser vendored by #55461: 0.57 MB, so **~4.8 MB**
### Size by feature (`.tar.gz`, versions from `cpp/thirdparty/versions.txt`
at the time of writing)
| Group | Dependency | Upstream | Trimmed |
|---|---|---:|---:|
| Already vendored (#55461) | xsimd 14.3.0 | 0.36 | 0.27 |
| | uriparser 1.0.2 | 0.21 | 0.08 |
| **Parquet** (`ARROW_PARQUET` turns on thrift and simdjson; thrift turns on
zlib and Boost) | Boost 1.92.0 | **136.15** | **4.92** |
| | thrift 0.25.0 | 4.83 | 0.94 |
| | simdjson 4.6.4 | 6.37 | 0.50 |
| | zlib 1.3.1 | 1.44 | 0.47 |
| Snappy (in practice needed to read most Parquet files) | snappy 1.2.2 |
1.06 | 0.08 |
| Other codecs | zstd 1.5.7 | 2.32 | 0.88 |
| | brotli 1.2.1 | 0.62 | 0.50 |
| | lz4 1.10.0 | 0.37 | 0.23 |
| | bzip2 1.0.8 | 0.77 | 0.09 |
| Compute / JSON / allocator | re2 2023-03-01 | 0.37 | 0.30 |
| | utf8proc 2.10.0 | 0.19 | 0.18 |
| | rapidjson | 1.06 | 0.15 |
| | mimalloc 3.4.1 | 1.35 | 0.35 |
| **Total** | | **157.47** | **9.94** |
Dataset and Acero need no further thirdparty dependencies. abseil isn't
needed: the pinned re2 (2023-03-01) is older than re2's abseil dependency.
Resulting package size, including the 4.2 MB package itself:
| Option | Approx. package size |
|---|---:|
| Today (minimal offline build) | 4.2 MB |
| Trimmed Parquet + Snappy (+ xsimd/uriparser) | **~11.5 MB** |
| Trimmed full non-cloud set (smallest full-featured option) | **~14 MB** |
| Upstream tarballs, only Boost trimmed | ~30 MB |
| Upstream tarballs, nothing trimmed | ~162 MB |
Boost is the only dependency that must be trimmed. Trimming the others saves
~16 MB.
### Trimming steps
**Boost (136 MB to 4.9 MB)**
1. Configure Boost standalone with the `BOOST_INCLUDE_LIBRARIES` list from
`build_boost()` in `ThirdpartyToolchain.cmake`. For a threaded build with
Parquet, that's
`algorithm;crc;numeric/conversion;scope_exit;throw_exception;tokenizer;locale;thread;uuid`.
Boost's CMake resolves the transitive dependencies, which come to 54
libraries. The libraries it adds appear as subdirectories of `<build>/libs`.
2. Keep only the top-level files, `tools/cmake`, and those 54 `libs/*`
directories.
3. Delete top-level `doc/`, `status/`, `more/`, and everything in `tools/`
except `tools/cmake`.
4. Inside each kept library, delete `test`, `doc`, `example(s)`,
`bench(mark)`, `perf`, `performance`, `meta` and `tools` directories.
If `multiprecision` is added (only needed when the compiler has no native
`int128`, see `ARROW_BOOST_NEED_MULTIPRECISION`), the trimmed tarball is 10.5
MB instead. gcc/clang/mingw all have native int128, so the smaller list should
be enough for R. The library list depends on Arrow's CMake configuration, so it
needs re-checking whenever Boost or `build_boost()` changes. The old
`cpp/build-support/trim-boost.sh` (removed in #50885) did something similar for
the b2-based Boost tarball.
**Everything else**
The generic step deletes `test`, `tests`, `testdata`, `doc`, `docs`,
`example(s)`, `benchmark(s)`, `bench` and `fuzz` directories. On top of that:
| Dependency | Extra removals | Keep |
|---|---|---|
| thrift | `tutorial/ test/ compiler/ contrib/ doc/ debian/`, all of `lib/`
except `lib/cpp` and `lib/CMakeLists.txt` (Arrow builds with
`BUILD_COMPILER=OFF` and `WITH_*=OFF` except C++) | |
| simdjson | `singleheader/` (20 MB uncompressed), `images/ jsonexamples/
tools/ scripts/` | |
| zlib | `contrib/` | `test/`, because zlib's CMake builds `test/example.c`
unless `ZLIB_BUILD_EXAMPLES=OFF` |
| brotli | `go/ python/ java/ csharp/ js/ research/` | `docs/`, because
`install()` copies the man pages |
| zstd | `contrib/ zlibWrapper/`, all of `build/` except `build/cmake` | |
| lz4 | all of `build/` except `build/cmake` | |
| bzip2 | `manual.*`, `sample*`, `*.html` | |
| re2 | `benchlog/ python/ app/` | |
| rapidjson | `bin/ thirdparty/` | |
| mimalloc | `ide/` | `bin/` (prebuilt Windows redirect DLLs; probably
unneeded for static builds, but not checked) |
| snappy, utf8proc, xsimd, uriparser | generic step only | |
Each trimmed tarball no longer matches the pinned
`ARROW_*_BUILD_SHA256_CHECKSUM` in `versions.txt`, and the bundled build checks
those hashes (`URL_HASH`). The R package has its own copy of the file, so
`sync-cpp` would rewrite the checksums in `r/tools/cpp/thirdparty/versions.txt`
to match the trimmed tarballs, the same way it already seds
`tools/cpp/CMakeLists.txt`. (Alternatively, trim only Boost and ship the rest
as-is: ~30 MB package.)
The scripts I used (directory names hard-code the current versions):
<details>
<summary>trim-boost.sh</summary>
```bash
#!/usr/bin/env bash
# Trim the Boost CMake source tarball to the libraries libarrow's bundled
# Boost build uses.
# Usage: trim-boost.sh boost-1.92.0-cmake.tar.gz OUT.tar.gz
set -euo pipefail
IN="$(cd "$(dirname "$1")" && pwd)/$(basename "$1")"
OUT="$(cd "$(dirname "$2")" && pwd)/$(basename "$2")"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT
cd "$WORK"
tar xzf "$IN"
SRC="$(ls -d boost-*)"
SRC_ABS="$WORK/$SRC"
# BOOST_INCLUDE_LIBRARIES from build_boost() in ThirdpartyToolchain.cmake
for a
# threaded build with Parquet (thrift) and native int128. Add multiprecision
if
# ARROW_BOOST_NEED_MULTIPRECISION would be true.
LIBS="algorithm;crc;numeric/conversion;scope_exit;throw_exception;tokenizer;locale;thread;uuid"
# Let Boost's CMake resolve transitive dependencies; each library it adds
# shows up as a directory under build/libs.
cmake -S "$SRC" -B build -DBOOST_INCLUDE_LIBRARIES="$LIBS" > /dev/null
KEEP=$(cd build/libs && find . -maxdepth 2 -type d ! -name CMakeFiles | sed
's|^\./||' |
while read -r d; do [ -f "$SRC_ABS/libs/$d/CMakeLists.txt" ] && echo
"libs/$d" || true; done)
mkdir trimmed
(cd "$SRC" && tar cf - $(ls | grep -v '^libs$') $KEEP) | (mkdir
"trimmed/$SRC" && tar xf - -C "trimmed/$SRC")
cd "trimmed/$SRC"
rm -rf doc status more
(cd tools && ls | grep -v '^cmake$' | xargs rm -rf)
find libs -depth -type d \( -name test -o -name doc -o -name example -o
-name examples \
-o -name bench -o -name benchmark -o -name perf -o -name performance -o
-name meta \
-o -name tools \) -prune -exec rm -rf {} +
cd ..
tar czf "$OUT" "$SRC"
```
</details>
<details>
<summary>trim.sh (everything else)</summary>
```bash
#!/usr/bin/env bash
# Trim thirdparty source tarballs down to what libarrow's bundled build uses.
# Usage: trim.sh SRC_TARBALL_DIR OUT_DIR
set -euo pipefail
IN="$(cd "$1" && pwd)"
OUT="$(mkdir -p "$2" && cd "$2" && pwd)"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT
rmd() { for p in "$@"; do rm -rf "$p"; done; }
trim_generic() {
# dirs no bundled build of these deps compiles; pass names to keep as args
local keep=("$@")
find . -depth -type d \( -name test -o -name tests -o -name testdata \
-o -name doc -o -name docs -o -name example -o -name examples \
-o -name benchmark -o -name benchmarks -o -name bench -o -name fuzz \) \
-prune -print | while read -r d; do
for k in "${keep[@]+"${keep[@]}"}"; do [ "$d" = "./$k" ] && continue
2; done
rm -rf "$d"
done
}
pack() { # dir, output name
tar czf "$OUT/$2" "$1"
}
cd "$WORK"
for t in "$IN"/*; do tar xf "$t"; done
# thrift: only the C++ library is built (BUILD_COMPILER=OFF, WITH_*=OFF)
( cd thrift-0.25.0 && rmd tutorial test compiler contrib doc debian \
&& (cd lib && ls | grep -v -E '^(cpp|CMakeLists.txt)$' | xargs rm -rf)
&& trim_generic )
pack thrift-0.25.0 thrift-0.25.0.tar.gz
# simdjson: built from src/ + include/; singleheader/ is a 20 MB amalgamation
( cd simdjson-4.6.4 && rmd singleheader images jsonexamples tools scripts &&
trim_generic )
pack simdjson-4.6.4 simdjson-v4.6.4.tar.gz
# zlib builds test/example.c by default (ZLIB_BUILD_EXAMPLES isn't turned
off)
( cd zlib-1.3.1 && rmd contrib && trim_generic test )
pack zlib-1.3.1 zlib-1.3.1.tar.gz
( cd snappy-1.2.2 && trim_generic )
pack snappy-1.2.2 snappy-1.2.2.tar.gz
( cd lz4-1.10.0 && trim_generic && (cd build && ls | grep -v '^cmake$' |
xargs rm -rf) )
pack lz4-1.10.0 lz4-v1.10.0.tar.gz
( cd zstd-1.5.7 && rmd contrib zlibWrapper && trim_generic \
&& (cd build && ls | grep -v '^cmake$' | xargs rm -rf) )
pack zstd-1.5.7 zstd-1.5.7.tar.gz
# brotli installs man pages from docs/
( cd brotli-1.2.1 && rmd go python java csharp js research && trim_generic
docs )
pack brotli-1.2.1 brotli-v1.2.1.tar.gz
( cd bzip2-1.0.8 && rm -f manual.* sample* *.html && trim_generic )
pack bzip2-1.0.8 bzip2-1.0.8.tar.gz
( cd re2-2023-03-01 && rmd benchlog python app && trim_generic )
pack re2-2023-03-01 re2-2023-03-01.tar.gz
( cd utf8proc-2.10.0 && trim_generic )
pack utf8proc-2.10.0 utf8proc-v2.10.0.tar.gz
R=rapidjson-232389d4f1012dddec4ef84861face2d2ba85709
( cd $R && rmd bin thirdparty && trim_generic )
pack $R $R.tar.gz
( cd mimalloc-3.4.1 && rmd ide && trim_generic )
pack mimalloc-3.4.1 mimalloc-v3.4.1.tar.gz
( cd xsimd-14.3.0 && trim_generic )
pack xsimd-14.3.0 xsimd-14.3.0.tar.gz
( cd uriparser-1.0.2 && trim_generic )
tar cjf "$OUT/uriparser-1.0.2.tar.bz2" uriparser-1.0.2
```
</details>
### Verification
On macOS arm64 (Apple clang), I ran `r/inst/build_arrow_static.sh` with
`LIBARROW_MINIMAL=false ARROW_DEPENDENCY_SOURCE=BUNDLED ARROW_S3=OFF
ARROW_GCS=OFF ARROW_AZURE=OFF ARROW_JEMALLOC=OFF`. Every `ARROW_*_URL` pointed
at a trimmed local tarball, and the checksums in `versions.txt` were rewritten
to match. The FetchContent/ExternalProject logs confirm that every dependency
came from a local file. `libarrow`, `libarrow_compute`, `libarrow_acero`,
`libarrow_dataset`, `libparquet` and `libarrow_bundled_dependencies` all built
and installed.
Not yet tested: Linux, Windows (Rtools), and actually installing and
checking the R package.
### What would need to change
1. `tools/vendor_dependencies.sh`: vendor the dependencies above, apply the
trimming, and rewrite the checksums in `r/tools/cpp/thirdparty/versions.txt`.
2. `nixlibs.R`: offline with no `thirdparty_dependency_dir`, the build
currently calls `turn_off_all_optional_features()` (around L630-644). It should
instead turn off only the features whose tarballs aren't vendored. S3 and Azure
need care, because they default to ON on macOS. The existing
`set_thirdparty_urls()` / `create_package_with_all_dependencies()` code already
handles "build from a directory of tarballs" and could probably be reused.
3. LICENSE/NOTICE: add entries for the newly shipped sources (Boost Software
License, thrift, simdjson, the codecs, etc.).
### Other considerations
- **CRAN policy:** CRAN asks packages to be "of the minimum necessary size",
so a ~14 MB source tarball may need justifying.
- **Build time:** building thrift, Boost.Locale/Thread and Parquet adds
noticeable compile time to CRAN source builds.
### Cloud filesystems (numbers only, not proposed)
For reference, the cloud filesystems are much more expensive and also need
system libraries that an offline build can't get:
| Feature | Tarballs (upstream) | Rough trimmed estimate | Also needs |
|---|---|---|---|
| S3 | aws-sdk-cpp 77.7 MB, aws-lc 138.5 MB (Linux), s2n 4.9 MB, aws-c-*
~5.3 MB (~226 MB) | ~25 MB (SDK limited to `config core identity-management s3
sts transfer` = 2.1 MB; aws-lc without test vectors and `fuzz/` = 13 MB). Not
build-tested | system libcurl, OpenSSL |
| GCS | google-cloud-cpp 16.2 MB, nlohmann-json 9.2 MB, abseil 2.3 MB (~28
MB) | not measured | system libcurl, OpenSSL |
| Azure | azure-sdk-for-cpp 3.0 MB | not measured | system libcurl, libxml2,
OpenSSL |
### Component(s)
R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
