github-actions[bot] commented on code in PR #70260:
URL: https://github.com/apache/doris/pull/70260#discussion_r4481672149
##########
build-support/tests/test-build-hygiene-header-deps.sh:
##########
@@ -99,6 +107,69 @@ cp
"${BACKUP}/be/src/storage/index/inverted/query/spi/index_source.h" \
"${ROOT}/be/src/storage/index/inverted/query/spi/index_source.h"
# ---- RED: third-party ban (uint24.h must not include fmt) ----
+# CLucene-free roots: an angle include two hops away from a root.
+printf '#pragma once\n#include <CLucene/util/Misc.h>\n' >
"${ROOT}/${PROBE_INNER}"
+printf '#pragma once\n#include "util/zz_clucene_probe_inner.h"\n' >
"${ROOT}/${PROBE_OUTER}"
+printf '#include "util/zz_clucene_probe_outer.h"\n' \
+ >> "${ROOT}/be/src/storage/index/inverted/query/spi/index_source.h"
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "two-hop CLucene include expected exit 1, got ${EC}"
+echo "${OUT}" | grep -q \
+ "query/spi/index_source.h reaches CLucene through
util/zz_clucene_probe_inner.h" \
+ || fail "two-hop CLucene include not reported"
+echo "${OUT}" | grep -q -- "-> <CLucene/util/Misc.h>" \
+ || fail "CLucene closure failure lacks the include at the end of its chain"
+cp "${BACKUP}/be/src/storage/index/inverted/query/spi/index_source.h" \
+ "${ROOT}/be/src/storage/index/inverted/query/spi/index_source.h"
+rm -f "${ROOT}/${PROBE_INNER}" "${ROOT}/${PROBE_OUTER}"
+
+# A source file under a root is a root too. The leading newline matters: the
file does not end
+# with one.
+printf '\n#include <CLucene.h>\n' \
+ >> "${ROOT}/be/src/storage/index/inverted/query/engine/intersection.cpp"
+OUT="$(run_gate)"; EC=$?
+[ "${EC}" -eq 1 ] || fail "CLucene include in a root source expected exit 1,
got ${EC}"
+echo "${OUT}" | grep -q "query/engine/intersection.cpp includes <CLucene.h>" \
+ || fail "CLucene include in a root source not reported"
+cp "${BACKUP}/be/src/storage/index/inverted/query/engine/intersection.cpp" \
+ "${ROOT}/be/src/storage/index/inverted/query/engine/intersection.cpp"
+
+# An exception that no longer breaks the rule fails the gate.
+sed -i '/#include <CLucene/d' \
Review Comment:
[P2] Use portable in-place sed syntax for both injections
On macOS/BSD, `sed -i` requires a backup-suffix argument, so this invocation
and the new one at line 163 fail instead of editing their targets. The gate
then sees the unchanged files, making both expected-red cases (and
`tests/run.sh`) fail. Use `sed -i.bak` followed by removing the backup, as
`build-support/AGENTS.md` specifies.
##########
be/benchmark/analyzer_benchmark.md:
##########
@@ -0,0 +1,98 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Analyzer benchmark
+
+`benchmark_analyzer.hpp` measures the text analyzers on the path the SNII
writer takes for one row,
+and writes a digest of every row's tokens, so a change to an analyzer can be
checked for speed and
+for identical output against a baseline binary.
+
+## Build
+
+Build a release benchmark binary of only this header:
+
+```shell
+BUILD_TYPE=RELEASE DORIS_BENCHMARK_HEADER=benchmark_analyzer.hpp ./build.sh
--benchmark
+```
+
+The binary is `be/output/lib/benchmark_test`; copy it away before building the
other side. The
+baseline is the commit that added this benchmark, on the branch
`snii-analysis-bench-baseline`:
+build it the same way from that commit.
+
+## Corpora
+
+The corpora are `<name>.rows` files, each row a little-endian u32 length and
that many bytes.
+They are not in the repository; `analyzer_corpus.py` builds them and lists the
sources and
+licenses in its header. The comparisons use at least the mixed corpora
`http_request`,
+`otel_body`, `agent_input` and `prose_en`, and the real Chinese and Japanese
text `zh_wiki` and
+`ja_wiki`:
+
+```shell
+be/benchmark/analyzer_corpus.py wikipedia zh corpus/zh_wiki.rows 8
+be/benchmark/analyzer_corpus.py wikipedia ja corpus/ja_wiki.rows 8
+be/benchmark/analyzer_corpus.py ndjson documents-171998.json request
corpus/http_request.rows
+```
+
+## Environment
+
+```shell
+export DORIS_HOME=<source tree>
+export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
+export ANALYZER_BENCH_CORPUS_DIR=<directory of .rows files>
+export ANALYZER_BENCH_CORPUS_MB=8 # bytes used from each corpus file
+export ANALYZER_BENCH_DICT_DIR=<output/be/dict of a BE build>
+```
+
+The dictionary directory has the layout of a BE package: the jieba files at
the top, and `icu`,
+`ik`, `pinyin` and `kuromoji` below. Without `ANALYZER_BENCH_CORPUS_DIR` the
header registers
+nothing.
+
+The benchmarks are named `row/<analyzer>/<corpus>`, a new stream per row as
the SNII writer does,
+and `reuse/<analyzer>/<corpus>`, one stream across the rows.
+
+## Throughput
+
+Host load moves single measurements by 10 to 20 percent, so the two binaries
are compared in
+paired rounds and only ratios are reported:
+
+```shell
+be/benchmark/analyzer_ab.py --baseline baseline/benchmark_test --candidate
new/benchmark_test \
Review Comment:
[P2] Make the documented Python entry points executable
Both `analyzer_ab.py` and `analyzer_corpus.py` are added with mode `100644`,
so this command and the corpus-generation commands above fail with `Permission
denied` in a clean checkout. Set both scripts to `100755` or invoke them with
`python3` in the instructions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
