morningman opened a new pull request, #66650:
URL: https://github.com/apache/doris/pull/66650
### What problem does this PR solve?
Related PR: #66358
Problem Summary:
`sh build-thirdparty.sh arrow_adbc` fails with
```
-- Installing: .../thirdparty/installed/lib64/pkgconfig/adbc-driver-sqlite.pc
.../thirdparty/src/adbc_driver_flightsql does not exist.
```
`build-thirdparty.sh` passes the package names straight through to
`download-thirdparty.sh`, which narrows `TP_ARCHIVES` to exactly the named
entries.
`ARROW_ADBC_FLIGHTSQL` is a companion archive of `arrow_adbc` rather than a
package of
its own -- it has no build function, `build_arrow_adbc()` only copies the
prebuilt
driver out of it -- so its name never appears on a command line and the
narrowing drops
it. The wheel is then neither downloaded nor unpacked, and
`check_if_source_exist` kills
the build.
A full `build-thirdparty.sh` run is unaffected:
| invocation | result |
| --- | --- |
| `build-thirdparty.sh` (no package argument) | ok |
| `build-thirdparty.sh arrow_adbc` on a clean tree | **fails** |
| `build-thirdparty.sh arrow_adbc` twice in a row | first ok, **second
fails** |
| `build-thirdparty.sh --clean arrow_adbc` | **fails** |
The third row is the confusing one: `cleanup_package_source()` deletes the
unpacked
driver once `arrow_adbc` is built, and the narrowed download list never
restores it, so
the first run succeeding is exactly what makes the second one fail.
The first commit pulls the companion archive in at the narrowing site,
guarded on the
platforms that register it -- upstream publishes no prebuilt driver for
macOS or other
architectures, where `ARROW_ADBC_FLIGHTSQL_SOURCE` stays unset and the
archive is not
part of `TP_ARCHIVES` to begin with.
The second commit is unrelated to the failure above, but was noticed while
debugging it.
The thirdparty scripts carry a `#!/usr/bin/env bash` shebang, which is
ignored when they
are started as `sh build-thirdparty.sh`. On Debian and Ubuntu `/bin/sh` is
dash, which
parses none of the `[[ ]]`, arrays and here-strings these scripts are built
on -- and
does not stop at the first one, it runs a mangled version of the script.
They now
re-exec themselves under bash.
### Release note
None
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
<details>
<summary>Manual test details</summary>
Linux / x86_64, with both archives already in `thirdparty/src` so nothing is
downloaded.
**Targeted download, before and after**
```
# before (upstream master)
$ bash download-thirdparty.sh arrow_adbc
Download and build specified libs only: ARROW_ADBC
$ ls src/ | grep '^adbc_driver_flightsql$'
(nothing -- this is the state build_arrow_adbc() then dies on)
# after
$ bash download-thirdparty.sh arrow_adbc
Download and build specified libs only: ARROW_ADBC ARROW_ADBC_FLIGHTSQL
$ ls -la src/adbc_driver_flightsql/libadbc_driver_flightsql.so
-rwxr-xr-x 1 ... 33693080 ...
src/adbc_driver_flightsql/libadbc_driver_flightsql.so
```
**Full targeted build, run twice from a tree that only holds the archives**
```
$ sh build-thirdparty.sh arrow_adbc # RUN1_EXIT=0
$ sh build-thirdparty.sh arrow_adbc # RUN2_EXIT=0
$ ls installed/lib64/ | grep adbc
libadbc_driver_flightsql.so
libadbc_driver_jni.so
libadbc_driver_manager.a
...
```
**Edge cases**
```
$ bash download-thirdparty.sh arrow_adbc_flightsql
Download and build specified libs only: ARROW_ADBC_FLIGHTSQL #
companion does not drag its owner in
$ bash download-thirdparty.sh arrow_adbc arrow_adbc_flightsql
Download and build specified libs only: ARROW_ADBC ARROW_ADBC_FLIGHTSQL #
not duplicated
# with `uname -s` stubbed to Darwin, so ARROW_ADBC_FLIGHTSQL_SOURCE stays
unset
$ bash download-thirdparty.sh arrow_adbc
Download and build specified libs only: ARROW_ADBC #
companion correctly not added
```
Removing the `-n "${ARROW_ADBC_FLIGHTSQL_SOURCE}"` guard makes that last
case fail with
`Failed to download ` (empty archive name), so the guard is load bearing
rather than
decorative.
**Second commit, under a real dash** (`python:3.11-slim`, `/bin/sh` ->
`/usr/bin/dash`)
```
# before
$ sh build-thirdparty.sh -h ; echo $?
build-thirdparty.sh: 34: Bad substitution
build-thirdparty.sh: 42: [[: not found
build-thirdparty.sh: 72: [[: not found
build-thirdparty.sh: 114: [[: not found
build-thirdparty.sh: 120: Syntax error: redirection unexpected
2
# after -- re-execs under bash and reaches the real toolchain check
# (the container has no LDB toolchain, which is why it stops there)
$ sh build-thirdparty.sh -h ; echo $?
Target system: Linux; Target arch: x86_64
Python 3.11.15
Error: wrong directory DORIS_GCC_HOME=./..
1
```
Under bash the guard is a no-op: `BASH_VERSION` is set, so no re-exec
happens and
behaviour is unchanged.
</details>
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
