This is an automated email from the ASF dual-hosted git repository.
airborne12 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3325747be0c [fix](inverted index) Count only visible rows in BM25
statistics of merge-on-write tables (#69438)
3325747be0c is described below
commit 3325747be0cf3b77b2eb7b938018b83a16c50809
Author: guptashivam04 <[email protected]>
AuthorDate: Mon Oct 26 15:35:19 2026 +0530
[fix](inverted index) Count only visible rows in BM25 statistics of
merge-on-write tables (#69438)
### What problem does this PR solve?
Issue Number: None
Related PR: #69352 (BM25 statistics shared across the parallel scanners
of a tablet; this PR is its follow-up for replicas), #69313 (BM25 token
totals after CLucene compaction)
Problem Summary:
On a merge-on-write table, BM25 `score()` gives the same row a different
score on each replica of its tablet. A query picks one replica per
tablet, so with `replication_num = 3` a row has up to three scores.
Pagination over `ORDER BY score()` then repeats and skips rows from one
page to the next. Each replica is stable on its own (`use_fix_replica`
pins the scores), but the replicas disagree.
`CollectionStatistics::process_segment()` takes the document count, the
token count and each term's document frequency straight from every
segment's inverted index. Under merge-on-write, a row overwritten or
deleted by a later load stays in its segment's index, marked only in the
delete bitmap. A row written with the delete sign, which CDC deletes and
`DELETE` produce, stays as a normal row that queries filter out by
predicate. Both kinds are counted until compaction removes them, and
each replica compacts on its own schedule. So each replica counts a
different set of rows nobody can see.
Reproduction (3 BEs, `replication_num = 3`, 8 buckets, auto compaction
off): 8 loads, 3 rounds of overwriting a quarter of the rows, a
`DELETE`, and full rows inserted with `__DORIS_DELETE_SIGN__ = 1`. Then
a full compaction (`POST /api/compaction/run?compact_type=full`) of the
tablets on one BE only. On master with #69352, the query below changed
the score of all 3,799 matching rows from run to run: 1,900 distinct
scores instead of 950. With `use_fix_replica = 0/1/2` each replica was
stable, but the three disagreed.
```sql
SELECT id, score() FROM rt WHERE SEARCH('content:jobs OR content:try')
ORDER BY score() DESC LIMIT 100000;
```
The token total has a second, replica-independent problem: CLucene index
compaction (`inverted_index_compaction_enable`, on by default) copies
the source token totals even when it drops rows, which #69313 addresses.
On master, the same rows scored differently in a compacted table and in
a table freshly loaded with its visible rows.
Fix:
When the scan reads a delete bitmap (merge-on-write),
`CollectionStatistics` describes only the rows the scan returns:
- `OlapScanner` passes the scan's delete bitmap and read version
(`MowHiddenRows`) to `collect()`, or to
`TabletCollectionStatistics::copy_to()` for a split tablet. Every
scanner of a tablet reads with the same bitmap and version, so the
statistics shared by #69352 stay correct.
- For each segment, the hidden rows are the rows the delete bitmap marks
at the read version (`get_agg`, the same call the rowset reader uses),
plus the rows whose hidden `__DORIS_DELETE_SIGN__` column is set. Every
unique-key table has that column, and `DELETE` on a merge-on-write table
writes such rows unless `enable_mow_light_delete` is on. The delete-sign
column is read only for segments whose zone map shows a non-zero sign
(or cannot tell), so a segment without CDC deletes costs nothing extra.
The read follows the query's `enable_segment_cache` and
`enable_page_cache`.
- Document count: the index's document count minus the hidden rows.
- Term document frequency: `IndexSource::scoring_doc_freq_among()`
counts how many hidden rows a term's postings list. When the hidden rows
are much rarer than the term's postings, it leaps between them with
`seek_block`. Otherwise it decodes the postings once and walks the
hidden rows alongside them, with no allocation, and a term without
postings costs nothing. SNII overrides it so that reserved-namespace
scoring terms resolve as `scoring_doc_freq` resolves them.
- Token count: the sum of the document lengths the norms of the returned
non-NULL rows stand for, or the returned row count for an index without
norms. The exact length of a hidden row is not stored anywhere: CLucene
and SNII keep only the encoded norm (exact up to 40 tokens for CLucene,
up to 254 for SNII). Subtracting decoded lengths from the exact
`sumTotalTermFreq` would still leave replicas apart. A row's norm is the
same in every segment it is written to, so summing norms gives every
replica the same count. It is also the same quantity the BM25 tf
normalization already divides, `dl / avgdl`, with `dl` taken from the
same norm.
- A NULL row has no tokens, but its norm stands for one (CLucene's
default norm, SNII's `encode_norm(0)`). Its length is left out, using
the index's null bitmap, and it still counts as a document, as before.
So on CLucene a table whose texts all have at most 40 tokens, and none
of which is an empty string, scores exactly as a duplicate-key table
with the same rows.
- The statistics read CLucene norms into a buffer of their own, one
segment at a time, and keep none. Asking a CLucene reader for a field's
norms makes it keep them for the searcher's lifetime, uncharged to the
searcher cache, and the statistics visit every segment of the read set,
including those the scan prunes and never scores. Reading per segment
also avoids the copy of all segments' norms a multi-segment reader would
keep.
- What the statistics keep instead is small and kept with the cached
searcher: the sum of a field's non-NULL norm lengths, and the sum of the
hidden rows' lengths for the last few sets of hidden rows of a field
(queries at one read version hide the same rows). So queries at an
unchanged version read no norms; a segment's norms are read again only
when its hidden rows change. The hidden and NULL row ID lists the
statistics build while collecting are charged to the query's memory.
Why the token count has to change too: a variant that left out hidden
rows only from the document count and the document frequencies, keeping
today's token total, made the replicas agree only while every replica
compacted through inverted index compaction, which carries dropped rows'
token counts forward. With one replica compacted with
`inverted_index_compaction_enable = false` (the index rebuilt from
column data), every row again scored differently on that replica. Its
scores were also 12–31 % off a fresh load of the visible rows on
average, because its token total still counted the hidden rows.
A tablet whose rows are all hidden (every row deleted or overwritten,
not yet compacted) has no document a query can match. Its statistics
report zero documents instead of failing the query with "No data
available for SimilarityCollector".
Tables without a delete bitmap (duplicate key, merge-on-read, or
`skip_delete_bitmap`) collect the same statistics as before.
For every `score()` query, with or without a delete bitmap:
- The statistics reads follow the query's `disable_file_cache` and
`inverted_index_snii_read_no_write_file_cache`, as the scan's reads do.
They run before the scan, so they are the reads that fill the file
cache. CLucene reads norms without a per-read context, as scoring does.
- A CLucene searcher the statistics open drops the query's I/O context
once its meta is read, as the query path already does
(`create_index_searcher`), and a CLucene error while opening it fails
the query; before, it escaped the scanner thread uncaught and aborted
the backend. Before, a searcher the statistics put into the searcher
cache kept the opening query's I/O context, including its file cache
statistics pointer, for later queries to read through.
Not covered:
- Merge-on-read unique tables and duplicate tables with `DELETE`
predicates also count rows that compaction later drops. They need merge
or predicate evaluation to tell which rows are visible, and are left for
a follow-up.
- With `skip_delete_sign = true` or `show_hidden_columns = true`, rows
written with the delete sign are returned but still left out of the
statistics. Rows deleted by a delete predicate
(`enable_mow_light_delete`) are still counted, as the delete bitmap does
not mark them.
- A row indexed without tokens that is not NULL keeps a norm that cannot
be told from a one-token row's, so on a merge-on-write table it counts
as one token: an analyzed empty string, which both formats index like a
NULL but leave out of the null bitmap, and on SNII any text that yields
no token (for example `'!!!'`, or text without ASCII letters or digits
under the `english` parser), whose norm SNII stores as that of one
token. On CLucene such a text has a zero norm and counts as zero tokens.
Telling these rows apart would mean reading the column.
- On SNII, the sum of the norm lengths is not cached yet: each
merge-on-write query reads the segment's norms.
- Leaving NULL rows out relies on the index's null bitmap, which CLucene
index compaction carries over; if reading a source bitmap fails there,
the compacted index loses it (existing behaviour of the vendored CLucene
compaction).
- Statistics stay per tablet, as before.
Interaction with #69313: that PR makes the token total of a CLucene
index built by compaction exact. With this PR, merge-on-write scans no
longer read that total, so their scores do not depend on it. Scans of
other tables still do. The two changes are independent.
---
be/src/exec/scan/olap_scanner.cpp | 25 +-
.../inverted/clucene/clucene_index_reader.cpp | 26 +-
.../index/inverted/clucene/clucene_index_reader.h | 7 +
.../index/inverted/clucene/searcher_cache_value.h | 68 ++++
.../inverted/clucene/spi/clucene_index_source.cpp | 76 ++++
.../inverted/clucene/spi/clucene_index_source.h | 8 +
.../index/inverted/query/exec/index_source.cpp | 131 +++++++
.../index/inverted/query/spi/index_source.h | 14 +
.../inverted/reader/collection_statistics.cpp | 295 ++++++++++++++-
.../index/inverted/reader/collection_statistics.h | 75 +++-
.../inverted/snii/reader/snii_index_source.cpp | 18 +
.../index/inverted/snii/reader/snii_index_source.h | 2 +
be/test/exec/scan/olap_scanner_test.cpp | 17 +
.../query/engine/multi_segment_collector_test.cpp | 60 ++-
.../inverted/reader/collection_statistics_test.cpp | 420 ++++++++++++++++++++-
.../segment/inverted_index_searcher_cache_test.cpp | 74 ++++
...t_bm25_score_statistics_searcher_io_context.out | 103 +++++
.../data/inverted_index_p0/test_bm25_score.out | 2 +-
.../test_bm25_score_replica_statistics.out | 6 +
.../inverted_index_p0/test_score_filter_matrix.out | 72 ++--
...m25_score_statistics_searcher_io_context.groovy | 61 +++
.../test_bm25_score_replica_statistics.groovy | 176 +++++++++
22 files changed, 1666 insertions(+), 70 deletions(-)
diff --git a/be/src/exec/scan/olap_scanner.cpp
b/be/src/exec/scan/olap_scanner.cpp
index eb0d1604f60..89e69bc2e6b 100644
--- a/be/src/exec/scan/olap_scanner.cpp
+++ b/be/src/exec/scan/olap_scanner.cpp
@@ -159,6 +159,14 @@ io::IOContext
build_score_runtime_collection_io_context(RuntimeState* state, Rea
if (auto* query_ctx = state->get_query_ctx(); query_ctx != nullptr) {
io_ctx.remote_scan_cache_write_limiter =
query_ctx->remote_scan_cache_write_limiter();
}
+ // The statistics read the index before the scan does, so they follow the
query's file cache
+ // choices as the scan's reads do.
+ if (reader_type == ReaderType::READER_QUERY) {
+ const auto& query_options = state->query_options();
+ io_ctx.is_disposable = query_options.disable_file_cache;
+ io_ctx.inverted_index_snii_read_no_write_file_cache =
+ query_options.inverted_index_snii_read_no_write_file_cache;
+ }
return io_ctx;
}
@@ -287,15 +295,28 @@ Status OlapScanner::_prepare_impl() {
_state, _tablet_reader_params.reader_type,
tablet->file_cache_ttl_expiration_time(),
&_tablet_reader->mutable_stats()->file_cache_stats);
+ // A merge-on-write scan leaves out rows its segments still hold,
which replicas drop
+ // at different times, so the statistics count only the rows the scan
returns.
+ std::optional<MowHiddenRows> hidden_rows;
+ if (_tablet_reader_params.delete_bitmap != nullptr) {
+ const auto& query_options = _state->query_options();
+ hidden_rows = MowHiddenRows {
+ .delete_bitmap = _tablet_reader_params.delete_bitmap,
+ .version = _tablet_reader_params.version.second,
+ .use_segment_cache =
!query_options.__isset.enable_segment_cache ||
+ query_options.enable_segment_cache,
+ .use_page_cache = _state->enable_page_cache()};
+ }
+ const MowHiddenRows* hidden_rows_ptr = hidden_rows ? &*hidden_rows :
nullptr;
if (_tablet_collection_statistics != nullptr) {
RETURN_IF_ERROR(_tablet_collection_statistics->copy_to(
_state, _tablet_reader_params.tablet_schema,
- _tablet_reader_params.common_expr_ctxs_push_down, &io_ctx,
+ _tablet_reader_params.common_expr_ctxs_push_down, &io_ctx,
hidden_rows_ptr,
_tablet_reader_params.collection_statistics.get()));
} else {
RETURN_IF_ERROR(_tablet_reader_params.collection_statistics->collect(
_state, _tablet_reader_params.rs_splits,
_tablet_reader_params.tablet_schema,
- _tablet_reader_params.common_expr_ctxs_push_down,
&io_ctx));
+ _tablet_reader_params.common_expr_ctxs_push_down, &io_ctx,
hidden_rows_ptr));
}
}
diff --git a/be/src/storage/index/inverted/clucene/clucene_index_reader.cpp
b/be/src/storage/index/inverted/clucene/clucene_index_reader.cpp
index a250c5b88a1..6c569dafdf4 100644
--- a/be/src/storage/index/inverted/clucene/clucene_index_reader.cpp
+++ b/be/src/storage/index/inverted/clucene/clucene_index_reader.cpp
@@ -154,18 +154,15 @@ namespace doris::segment_v2 {
namespace logical = index_query::logical;
-namespace {
-
-Status read_clucene_null_bitmap(InvertedIndexReader& reader, const
IndexQueryContextPtr& context,
+Status read_clucene_null_bitmap(IndexFileReader& file_reader, const
TabletIndex& index_meta,
+ const io::IOContext* io_ctx,
InvertedIndexQueryCacheHandle* cache_handle,
lucene::store::Directory* dir) {
- SCOPED_RAW_TIMER(&context->stats->inverted_index_query_null_bitmap_timer);
lucene::store::IndexInput* null_bitmap_in = nullptr;
bool owned_dir = false;
try {
// try to get query bitmap result from cache and return immediately on
cache hit
- auto index_file_key =
-
reader.get_index_file_reader()->get_index_file_cache_key(&reader.get_index_meta());
+ auto index_file_key =
file_reader.get_index_file_cache_key(&index_meta);
InvertedIndexQueryCache::CacheKey cache_key {
index_file_key, "", InvertedIndexQueryType::UNKNOWN_QUERY,
"null_bitmap"};
auto* cache = InvertedIndexQueryCache::instance();
@@ -175,14 +172,12 @@ Status read_clucene_null_bitmap(InvertedIndexReader&
reader, const IndexQueryCon
if (!dir) {
// TODO: ugly code here, try to refact.
- auto st =
reader.get_index_file_reader()->init(config::inverted_index_read_buffer_size,
- context->io_ctx);
+ auto st =
file_reader.init(config::inverted_index_read_buffer_size, io_ctx);
if (!st.ok()) {
LOG(WARNING) << st;
return st;
}
- auto directory = DORIS_TRY(reader.get_index_file_reader()->open(
- &reader.get_index_meta(), context->io_ctx));
+ auto directory = DORIS_TRY(file_reader.open(&index_meta, io_ctx));
dir = directory.release();
owned_dir = true;
}
@@ -219,6 +214,17 @@ Status read_clucene_null_bitmap(InvertedIndexReader&
reader, const IndexQueryCon
return Status::OK();
}
+namespace {
+
+Status read_clucene_null_bitmap(InvertedIndexReader& reader, const
IndexQueryContextPtr& context,
+ InvertedIndexQueryCacheHandle* cache_handle,
+ lucene::store::Directory* dir) {
+ SCOPED_RAW_TIMER(&context->stats->inverted_index_query_null_bitmap_timer);
+ return
segment_v2::read_clucene_null_bitmap(*reader.get_index_file_reader(),
+ reader.get_index_meta(),
context->io_ctx,
+ cache_handle, dir);
+}
+
// NOLINTNEXTLINE(readability-function-cognitive-complexity) Cache lookup and
opening share this flow.
Status handle_clucene_searcher_cache(InvertedIndexReader& reader,
const IndexQueryContextPtr& context,
diff --git a/be/src/storage/index/inverted/clucene/clucene_index_reader.h
b/be/src/storage/index/inverted/clucene/clucene_index_reader.h
index 2b4d2fd4c1b..6319f2fc439 100644
--- a/be/src/storage/index/inverted/clucene/clucene_index_reader.h
+++ b/be/src/storage/index/inverted/clucene/clucene_index_reader.h
@@ -51,6 +51,13 @@ protected:
std::shared_ptr<roaring::Roaring>* out) override;
};
+// The null bitmap of a CLucene index, through the inverted index query cache.
Reads it from
+// `dir` when that is the index's open directory, otherwise opens the index.
+Status read_clucene_null_bitmap(IndexFileReader& file_reader, const
TabletIndex& index_meta,
+ const io::IOContext* io_ctx,
+ InvertedIndexQueryCacheHandle* cache_handle,
+ lucene::store::Directory* dir = nullptr);
+
// run_leaf on the CLucene field `field` of `searcher`, its errors reported as
CLucene ones.
Status run_clucene_leaf(const IndexQueryContextPtr& context, std::wstring
field,
const index_query::logical::Node& leaf, const
roaring::Roaring* candidates,
diff --git a/be/src/storage/index/inverted/clucene/searcher_cache_value.h
b/be/src/storage/index/inverted/clucene/searcher_cache_value.h
index 0da255ee6b3..8d330f86ee6 100644
--- a/be/src/storage/index/inverted/clucene/searcher_cache_value.h
+++ b/be/src/storage/index/inverted/clucene/searcher_cache_value.h
@@ -17,10 +17,19 @@
#pragma once
+#include <cstdint>
+#include <functional>
+#include <mutex>
+#include <span>
+#include <string>
+#include <unordered_map>
#include <utility>
+#include <vector>
+#include "common/status.h"
#include "storage/index/inverted/clucene/inverted_index_searcher.h"
#include "storage/index/inverted/common/inverted_index_cache.h"
+#include "util/hash_util.hpp"
namespace doris::segment_v2 {
@@ -28,7 +37,66 @@ struct CluceneSearcherCacheValue final :
InvertedIndexSearcherCache::CacheValue
CluceneSearcherCacheValue(IndexSearcherPtr opened, size_t mem_size,
int64_t visit_time)
: CacheValue(mem_size, visit_time), searcher(std::move(opened)) {}
+ // The sum of the lengths the norms of the non-NULL documents of `field`
stand for, computed
+ // by `compute` the first time. The index never changes, and
merge-on-write scoring needs the
+ // sum for every query.
+ Status non_null_norm_length_sum(const std::wstring& field,
+ const std::function<Status(uint64_t*)>&
compute,
+ uint64_t* out) {
+ std::lock_guard lock(_norm_length_sums_mutex);
+ if (auto it = _norm_length_sums.find(field); it !=
_norm_length_sums.end()) {
+ *out = it->second;
+ return Status::OK();
+ }
+ RETURN_IF_ERROR(compute(out));
+ _norm_length_sums.emplace(field, *out);
+ return Status::OK();
+ }
+
+ // The sum of the lengths the norms of `docs`, ascending rows of the
index, stand for, computed
+ // by `compute` the first time the field meets that set of rows. Queries
at one read version
+ // hide the same rows, so they read no norms; a few sets per field are
kept for queries at
+ // different versions.
+ Status hidden_norm_length_sum(const std::wstring& field, std::span<const
uint32_t> docs,
+ const std::function<Status(uint64_t*)>&
compute, uint64_t* out) {
+ const RowSetKey key {
+ .fingerprint = HashUtil::xxHash64WithSeed(
+ reinterpret_cast<const char*>(docs.data()),
docs.size_bytes(), 0),
+ .count = docs.size(),
+ .first = docs.empty() ? 0 : docs.front(),
+ .last = docs.empty() ? 0 : docs.back()};
+ std::lock_guard lock(_norm_length_sums_mutex);
+ auto& sums = _hidden_length_sums[field];
+ for (const auto& [cached_key, sum] : sums) {
+ if (cached_key == key) {
+ *out = sum;
+ return Status::OK();
+ }
+ }
+ RETURN_IF_ERROR(compute(out));
+ if (sums.size() == kHiddenSetsPerField) {
+ sums.erase(sums.begin());
+ }
+ sums.emplace_back(key, *out);
+ return Status::OK();
+ }
+
IndexSearcherPtr searcher;
+
+private:
+ struct RowSetKey {
+ uint64_t fingerprint;
+ size_t count;
+ uint32_t first;
+ uint32_t last;
+ bool operator==(const RowSetKey&) const = default;
+ };
+ static constexpr size_t kHiddenSetsPerField = 4;
+
+ std::mutex _norm_length_sums_mutex;
+ std::unordered_map<std::wstring, uint64_t> _norm_length_sums;
+ std::unordered_map<std::wstring, std::vector<std::pair<RowSetKey,
uint64_t>>>
+ _hidden_length_sums;
};
} // namespace doris::segment_v2
diff --git a/be/src/storage/index/inverted/clucene/spi/clucene_index_source.cpp
b/be/src/storage/index/inverted/clucene/spi/clucene_index_source.cpp
index c509c07a2a6..81f9451e19f 100644
--- a/be/src/storage/index/inverted/clucene/spi/clucene_index_source.cpp
+++ b/be/src/storage/index/inverted/clucene/spi/clucene_index_source.cpp
@@ -243,6 +243,82 @@ std::span<const float> CluceneIndexSource::norm_lengths()
const {
return BM25Similarity::lucene_norm_lengths();
}
+// The statistics read a field's norms into a buffer of their own, freed after
use: asking a reader
+// for its norms makes it keep them, uncharged to the searcher cache, for
every segment the
+// statistics visit, scored or not. Reading per segment also avoids the copy
of all segments' norms
+// a reader of several segments would keep. A segment without the field's
norms reads as zero
+// lengths, as that reader would.
+std::vector<CluceneIndexSource::ReaderPartition>
CluceneIndexSource::_norm_partitions() const {
+ if (!_partitions.empty()) {
+ return _partitions;
+ }
+ return {{.reader = _reader.get(), .begin = 0, .end = doc_count()}};
+}
+
+void CluceneIndexSource::_read_norms(const ReaderPartition& partition,
+ DorisVector<uint8_t>* norms) const {
+ norms->assign(partition.end - partition.begin, 0);
+ partition.reader->norms(_field.c_str(), norms->data());
+}
+
+Status CluceneIndexSource::sum_norm_lengths(uint64_t* out) {
+ *out = 0;
+ try {
+ if (!_reader->hasNorms(_field.c_str())) {
+ return Status::Error<ErrorCode::INVERTED_INDEX_NOT_SUPPORTED>(
+ "field {} has no norms",
inverted_index::StringHelper::to_string(_field));
+ }
+ const auto lengths = norm_lengths();
+ DorisVector<uint8_t> norms;
+ for (const ReaderPartition& partition : _norm_partitions()) {
+ _read_norms(partition, &norms);
+ for (const uint8_t norm : norms) {
+ *out += static_cast<uint64_t>(lengths[norm]);
+ }
+ }
+ } catch (CLuceneError& e) {
+ return clucene_error_status(e.what());
+ }
+ return Status::OK();
+}
+
+Status CluceneIndexSource::sum_norm_lengths_of(std::span<const uint32_t> docs,
uint64_t* out) {
+ *out = 0;
+ if (docs.empty()) {
+ return Status::OK();
+ }
+ try {
+ if (!_reader->hasNorms(_field.c_str())) {
+ return Status::Error<ErrorCode::INVERTED_INDEX_NOT_SUPPORTED>(
+ "field {} has no norms",
inverted_index::StringHelper::to_string(_field));
+ }
+ const auto lengths = norm_lengths();
+ const uint32_t count = doc_count();
+ if (docs.back() >= count) {
+ return Status::Corruption("document {} is beyond the {} documents
of the index",
+ docs.back(), count);
+ }
+ DorisVector<uint8_t> norms;
+ // `docs` ascend, so each partition's documents are contiguous in them.
+ size_t next = 0;
+ for (const ReaderPartition& partition : _norm_partitions()) {
+ if (next == docs.size()) {
+ break;
+ }
+ if (docs[next] >= partition.end) {
+ continue;
+ }
+ _read_norms(partition, &norms);
+ for (; next < docs.size() && docs[next] < partition.end; ++next) {
+ *out += static_cast<uint64_t>(lengths[norms[docs[next] -
partition.begin]]);
+ }
+ }
+ } catch (CLuceneError& e) {
+ return clucene_error_status(e.what());
+ }
+ return Status::OK();
+}
+
bool CluceneIndexSource::is_live(uint32_t doc) const {
return !_reader->isDeleted(static_cast<int32_t>(doc));
}
diff --git a/be/src/storage/index/inverted/clucene/spi/clucene_index_source.h
b/be/src/storage/index/inverted/clucene/spi/clucene_index_source.h
index a0ae497ad51..c7aa7739042 100644
--- a/be/src/storage/index/inverted/clucene/spi/clucene_index_source.h
+++ b/be/src/storage/index/inverted/clucene/spi/clucene_index_source.h
@@ -25,6 +25,7 @@
#include <vector>
#include "common/status.h"
+#include "core/custom_allocator.h"
#include "storage/index/inverted/query/spi/index_source.h"
namespace lucene::index {
@@ -53,6 +54,8 @@ public:
Status expand_terms(index_query::TermPattern& pattern, int32_t
max_expansions,
std::vector<std::string>* out) override;
std::span<const float> norm_lengths() const override;
+ Status sum_norm_lengths(uint64_t* out) override;
+ Status sum_norm_lengths_of(std::span<const uint32_t> docs, uint64_t* out)
override;
bool is_live(uint32_t doc) const override;
lucene::index::IndexReader* reader() const { return _reader.get(); }
@@ -66,6 +69,11 @@ private:
};
void _append_partitions(lucene::index::IndexReader* reader, uint32_t
begin);
+ // The partitions whose norms make up the field's, the whole reader when
it has no others.
+ std::vector<ReaderPartition> _norm_partitions() const;
+ // Reads a partition's encoded norms of the field, one per document, into
`norms`, without the
+ // reader keeping them.
+ void _read_norms(const ReaderPartition& partition, DorisVector<uint8_t>*
norms) const;
std::shared_ptr<lucene::index::IndexReader> _reader;
std::wstring _field;
diff --git a/be/src/storage/index/inverted/query/exec/index_source.cpp
b/be/src/storage/index/inverted/query/exec/index_source.cpp
index 9ba0b73aa6c..8ad6c6b4184 100644
--- a/be/src/storage/index/inverted/query/exec/index_source.cpp
+++ b/be/src/storage/index/inverted/query/exec/index_source.cpp
@@ -17,6 +17,9 @@
#include "storage/index/inverted/query/spi/index_source.h"
+#include <algorithm>
+#include <numeric>
+
#include "storage/index/inverted/query/exec/cursor_chained_postings.h"
#include "storage/index/inverted/query/exec/docid_sink.h"
#include "storage/index/inverted/query/exec/term_waves.h"
@@ -43,4 +46,132 @@ Status IndexSource::collect_terms(std::span<const
std::string> terms, DocIdSink&
return Status::OK();
}
+namespace {
+
+// How many postings a scan decodes in the time a seek costs.
+constexpr uint64_t kPostingsPerSeek = 32;
+
+// Decodes every posting once and walks `docs` alongside, both ascending.
+Status count_listed_docs_by_scan(PostingsCursor& cursor, std::span<const
uint32_t> docs,
+ uint64_t* out) {
+ size_t next = 0;
+ PostingsBlock block;
+ bool eof = false;
+ while (next < docs.size()) {
+ RETURN_IF_ERROR(cursor.next_block(&block, &eof));
+ if (eof) {
+ break;
+ }
+ if (block.dense) {
+ const auto begin = std::lower_bound(docs.begin() + next,
docs.end(), block.range_begin);
+ const auto end = std::lower_bound(begin, docs.end(),
block.range_end);
+ *out += static_cast<uint64_t>(end - begin);
+ next = end - docs.begin();
+ continue;
+ }
+ for (const uint32_t doc : block.docs) {
+ while (next < docs.size() && docs[next] < doc) {
+ ++next;
+ }
+ if (next == docs.size()) {
+ break;
+ }
+ if (docs[next] == doc) {
+ ++*out;
+ ++next;
+ }
+ }
+ }
+ return Status::OK();
+}
+
+} // namespace
+
+Status count_listed_docs(PostingsCursor& cursor, std::span<const uint32_t>
docs, uint64_t* out) {
+ *out = 0;
+ if (docs.empty() || cursor.doc_freq() == 0) {
+ return Status::OK();
+ }
+ // Leaping seeks once per wanted document, scanning decodes every posting:
leap only over
+ // documents much rarer than the postings.
+ if (docs.size() * kPostingsPerSeek >= cursor.doc_freq()) {
+ return count_listed_docs_by_scan(cursor, docs, out);
+ }
+ size_t next = 0;
+ PostingsBlock block;
+ bool eof = false;
+ while (next < docs.size()) {
+ RETURN_IF_ERROR(cursor.seek_block(docs[next], &block, &eof));
+ if (eof) {
+ break;
+ }
+ const uint64_t size = block.size();
+ uint64_t ordinal = 0;
+ while (ordinal < size && next < docs.size()) {
+ const uint32_t listed = block.doc_at(ordinal);
+ const uint32_t wanted = docs[next];
+ if (listed == wanted) {
+ ++*out;
+ ++ordinal;
+ ++next;
+ } else if (listed < wanted) {
+ if (block.dense) {
+ ordinal = wanted - block.range_begin;
+ } else {
+ ordinal = std::lower_bound(block.docs.begin() + ordinal,
block.docs.end(),
+ wanted) -
+ block.docs.begin();
+ }
+ } else {
+ next = std::lower_bound(docs.begin() + next, docs.end(),
listed) - docs.begin();
+ }
+ }
+ }
+ return Status::OK();
+}
+
+Status IndexSource::scoring_doc_freq_among(std::string_view term,
std::span<const uint32_t> docs,
+ uint64_t* out) {
+ *out = 0;
+ if (docs.empty()) {
+ return Status::OK();
+ }
+ std::unique_ptr<PostingsCursor> cursor;
+ RETURN_IF_ERROR(open_term(term, /*positions=*/false, /*scoring=*/false,
&cursor));
+ if (cursor == nullptr) {
+ return Status::OK();
+ }
+ return count_listed_docs(*cursor, docs, out);
+}
+
+Status IndexSource::sum_norm_lengths_of(std::span<const uint32_t> docs,
uint64_t* out) {
+ *out = 0;
+ const std::span<const float> lengths = norm_lengths();
+ std::vector<uint32_t> norms;
+ RETURN_IF_ERROR(encoded_norms(docs, &norms));
+ for (const uint32_t norm : norms) {
+ if (norm >= lengths.size()) {
+ return Status::Corruption("encoded norm {} is out of range", norm);
+ }
+ *out += static_cast<uint64_t>(lengths[norm]);
+ }
+ return Status::OK();
+}
+
+Status IndexSource::sum_norm_lengths(uint64_t* out) {
+ *out = 0;
+ constexpr uint32_t kBatch = 4096;
+ std::vector<uint32_t> docs;
+ docs.reserve(kBatch);
+ const uint32_t count = doc_count();
+ for (uint32_t begin = 0; begin < count; begin += std::min(kBatch, count -
begin)) {
+ docs.resize(std::min(kBatch, count - begin));
+ std::iota(docs.begin(), docs.end(), begin);
+ uint64_t batch = 0;
+ RETURN_IF_ERROR(sum_norm_lengths_of(docs, &batch));
+ *out += batch;
+ }
+ return Status::OK();
+}
+
} // namespace doris::index_query
diff --git a/be/src/storage/index/inverted/query/spi/index_source.h
b/be/src/storage/index/inverted/query/spi/index_source.h
index 097ef06a1b1..991969b92c7 100644
--- a/be/src/storage/index/inverted/query/spi/index_source.h
+++ b/be/src/storage/index/inverted/query/spi/index_source.h
@@ -53,6 +53,10 @@ inline constexpr std::array<float, 256> kByteNormLengths =
[] {
return lengths;
}();
+// How many of the ascending `docs` the cursor lists: leaping over the
postings between them when
+// the docs are much rarer than the postings, otherwise decoding every posting
once.
+Status count_listed_docs(PostingsCursor& cursor, std::span<const uint32_t>
docs, uint64_t* out);
+
// One field's index as the engine reads it: postings by term, the dictionary
by pattern and
// the documents still alive. Every source uses the Doris segment's document
IDs.
class IndexSource {
@@ -92,6 +96,10 @@ public:
return doc_freq(term, out);
}
+ // How many of the ascending `docs` hold a term, the term read as
scoring_doc_freq reads it.
+ virtual Status scoring_doc_freq_among(std::string_view term,
std::span<const uint32_t> docs,
+ uint64_t* out);
+
// The terms `pattern` matches, in dictionary order, at most
`max_expansions` of them when
// that is positive.
virtual Status expand_terms(TermPattern& pattern, int32_t max_expansions,
@@ -108,6 +116,12 @@ public:
return Status::OK();
}
+ // The sum of the lengths norm_lengths() gives the encoded norms of every
document.
+ virtual Status sum_norm_lengths(uint64_t* out);
+
+ // The sum of the lengths norm_lengths() gives the encoded norms of the
ascending `docs`.
+ virtual Status sum_norm_lengths_of(std::span<const uint32_t> docs,
uint64_t* out);
+
// Whether the source reads in batched rounds: a leaf then opens all its
terms at once and
// the engine lists them term at a time, materializing what it needs,
instead of driving
// one document at a time across them.
diff --git a/be/src/storage/index/inverted/reader/collection_statistics.cpp
b/be/src/storage/index/inverted/reader/collection_statistics.cpp
index 4cf3b8d287b..11a1a5d4bb0 100644
--- a/be/src/storage/index/inverted/reader/collection_statistics.cpp
+++ b/be/src/storage/index/inverted/reader/collection_statistics.cpp
@@ -17,25 +17,41 @@
#include "storage/index/inverted/reader/collection_statistics.h"
+#include <algorithm>
#include <exception>
+#include <iterator>
#include <set>
#include <sstream>
#include "common/exception.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type_factory.hpp"
#include "exprs/vexpr.h"
#include "exprs/vexpr_context.h"
#include "exprs/vliteral.h"
#include "exprs/vslot_ref.h"
+#include "runtime/thread_context.h"
#include "storage/index/index_file_reader.h"
#include "storage/index/index_reader_helper.h"
#include "storage/index/inverted/analysis/analyzer/analyzer.h"
+#include "storage/index/inverted/clucene/clucene_index_reader.h"
#include "storage/index/inverted/clucene/searcher_cache_value.h"
#include "storage/index/inverted/clucene/spi/clucene_index_source.h"
+#include "storage/index/inverted/common/inverted_index_cache.h"
#include "storage/index/inverted/common/string_helper.h"
#include "storage/index/inverted/snii/reader/snii_index_source.h"
#include "storage/index/inverted/snii/snii_doris_adapter.h"
+#include "storage/iterators.h"
+#include "storage/olap_common.h"
+#include "storage/rowset/beta_rowset.h"
#include "storage/rowset/rowset.h"
#include "storage/rowset/rowset_reader.h"
+#include "storage/segment/column_reader.h"
+#include "storage/segment/segment.h"
+#include "storage/segment/segment_loader.h"
+#include "storage/tablet/tablet_meta.h"
+#include "util/defer_op.h"
#include "util/uid_util.h"
namespace doris {
@@ -55,7 +71,19 @@ Result<std::shared_ptr<lucene::search::IndexSearcher>>
open_optional_clucene_sta
}
return ResultError(status);
}
- auto* reader = lucene::index::IndexReader::open(compound.value().get());
+ lucene::index::IndexReader* reader = nullptr;
+ try {
+ reader = lucene::index::IndexReader::open(compound.value().get());
+ } catch (const CLuceneError& e) {
+ return
ResultError(Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>(
+ "CLuceneError occurred opening the index for BM25 statistics:
{}", e.what()));
+ }
+ // As the query path does once the index meta is read: the searcher may be
cached and read
+ // by later queries, which must not use this query's I/O context, and
later reads are of
+ // index data, not meta.
+ auto* stream = compound.value()->getDorisIndexInput();
+ stream->setIoContext(nullptr);
+ stream->setIndexFile(false);
return std::make_shared<lucene::search::IndexSearcher>(reader, true);
}
@@ -89,13 +117,175 @@ void add_term_doc_frequency(
(*logical_frequencies)[field][logical_term] += doc_frequency;
}
+Status sum_non_null_norm_lengths(index_query::IndexSource& source,
+ std::span<const uint32_t> null_docs,
uint64_t* out) {
+ if (!null_docs.empty() && null_docs.back() >= source.doc_count()) {
+ return Status::Error<ErrorCode::INVERTED_INDEX_FILE_CORRUPTED>(
+ "NULL row {} is beyond the {} documents of the index",
null_docs.back(),
+ source.doc_count());
+ }
+ uint64_t all_lengths = 0;
+ RETURN_IF_ERROR(source.sum_norm_lengths(&all_lengths));
+ uint64_t null_lengths = 0;
+ RETURN_IF_ERROR(source.sum_norm_lengths_of(null_docs, &null_lengths));
+ DORIS_CHECK(null_lengths <= all_lengths);
+ *out = all_lengths - null_lengths;
+ return Status::OK();
+}
+
+Status exclude_hidden_docs(index_query::IndexSource& source, std::span<const
uint32_t> hidden_docs,
+ const roaring::Roaring& null_docs, uint64_t
non_null_length_sum,
+ index_query::IndexScoringStats* stats) {
+ return exclude_hidden_docs(
+ hidden_docs, null_docs, non_null_length_sum,
+ [&source](std::span<const uint32_t> docs, uint64_t* out) {
+ return source.sum_norm_lengths_of(docs, out);
+ },
+ stats);
+}
+
+Status exclude_hidden_docs(std::span<const uint32_t> hidden_docs, const
roaring::Roaring& null_docs,
+ uint64_t non_null_length_sum,
+ const SumNormLengthsOf& sum_norm_lengths_of,
+ index_query::IndexScoringStats* stats) {
+ DORIS_CHECK(stats != nullptr);
+ if (!hidden_docs.empty() && hidden_docs.back() >= stats->doc_count) {
+ return Status::Error<ErrorCode::INVERTED_INDEX_FILE_CORRUPTED>(
+ "hidden row {} is beyond the {} documents of the index",
hidden_docs.back(),
+ stats->doc_count);
+ }
+ stats->doc_count -= hidden_docs.size();
+ if (!stats->has_norms) {
+ // Without norms every document stands for one token.
+ stats->token_count = stats->doc_count;
+ return Status::OK();
+ }
+ // The exact token count of the hidden rows is not stored anywhere, and
norms hold lengths
+ // only approximately, so count the lengths the norms of the returned rows
stand for. A
+ // replica that has compacted the hidden rows away counts the same rows
with the same norms.
+ // Most segments have no NULL rows, and then the hidden rows are summed as
they are.
+ DorisVector<uint32_t> hidden_non_null_docs;
+ std::span<const uint32_t> summed_docs = hidden_docs;
+ if (!null_docs.isEmpty()) {
+ std::ranges::copy_if(hidden_docs,
std::back_inserter(hidden_non_null_docs),
+ [&null_docs](uint32_t doc) { return
!null_docs.contains(doc); });
+ summed_docs = hidden_non_null_docs;
+ }
+ uint64_t hidden_lengths = 0;
+ if (!summed_docs.empty()) {
+ RETURN_IF_ERROR(sum_norm_lengths_of(summed_docs, &hidden_lengths));
+ }
+ DORIS_CHECK(hidden_lengths <= non_null_length_sum);
+ stats->token_count = non_null_length_sum - hidden_lengths;
+ return Status::OK();
+}
+
+Status exclude_hidden_term_docs(index_query::IndexSource& source,
std::string_view term,
+ std::span<const uint32_t> hidden_docs,
uint64_t* doc_frequency) {
+ DORIS_CHECK(doc_frequency != nullptr);
+ uint64_t hidden = 0;
+ RETURN_IF_ERROR(source.scoring_doc_freq_among(term, hidden_docs, &hidden));
+ if (hidden > *doc_frequency) {
+ return Status::Error<ErrorCode::INVERTED_INDEX_FILE_CORRUPTED>(
+ "term lists {} hidden rows but only {} documents", hidden,
*doc_frequency);
+ }
+ *doc_frequency -= hidden;
+ return Status::OK();
+}
+
} // namespace collection_statistics_detail
+namespace {
+
+// The rows of a segment that a scan with `hidden_rows` does not return,
ascending.
+Status collect_hidden_docs(const RowsetSharedPtr& rowset, const
RowsetSegmentView& seg,
+ const TabletSchema& tablet_schema, const
MowHiddenRows& hidden_rows,
+ const io::IOContext* io_ctx, DorisVector<uint32_t>*
hidden_docs) {
+ DORIS_CHECK(hidden_rows.delete_bitmap != nullptr);
+ DORIS_CHECK(hidden_rows.version >= 0);
+ roaring::Roaring hidden =
+
*hidden_rows.delete_bitmap->get_agg(seg.delete_bitmap_key(hidden_rows.version));
+
+ // A row written with the delete sign stays readable until base compaction
removes it, while
+ // queries filter it out by predicate.
+ if (tablet_schema.delete_sign_idx() >= 0) {
+ OlapReaderStatistics stats;
+ SegmentCacheHandle cache_handle;
+ // The delete sign is column data, read as the query reads its columns.
+ io::IOContext data_io_ctx;
+ if (io_ctx != nullptr) {
+ data_io_ctx = *io_ctx;
+ }
+ data_io_ctx.is_inverted_index = false;
+ RETURN_IF_ERROR(SegmentLoader::instance()->load_segment(
+ std::static_pointer_cast<BetaRowset>(rowset), seg.ref(),
&cache_handle,
+ hidden_rows.use_segment_cache, false, &stats, &data_io_ctx));
+ DORIS_CHECK(cache_handle.get_segments().size() == 1);
+ const auto& segment = cache_handle.get_segments()[0];
+ const TabletColumn& delete_sign =
tablet_schema.column(tablet_schema.delete_sign_idx());
+
+ StorageReadOptions read_options(stats);
+ read_options.version = rowset->version();
+ read_options.io_ctx = data_io_ctx;
+ std::shared_ptr<segment_v2::ColumnReader> column_reader;
+ RETURN_IF_ERROR(
+ segment->get_column_reader_for_pruning(delete_sign,
read_options, &column_reader));
+ bool may_hold_deletes = true;
+ if (column_reader->has_zone_map()) {
+ segment_v2::ZoneMap zone_map;
+ RETURN_IF_ERROR(column_reader->get_segment_zone_map(&zone_map));
+ // Any sign other than 0 deletes the row. Bounds the zone map
cannot vouch for say
+ // nothing.
+ may_hold_deletes = zone_map.pass_all || (zone_map.has_not_null &&
+
(zone_map.min_value.get<TYPE_TINYINT>() != 0 ||
+
zone_map.max_value.get<TYPE_TINYINT>() != 0));
+ }
+ if (may_hold_deletes) {
+ std::unique_ptr<segment_v2::ColumnIterator> iterator;
+ RETURN_IF_ERROR(segment->new_column_iterator(delete_sign,
&iterator, &read_options));
+ segment_v2::ColumnIteratorOptions iterator_options {
+ .use_page_cache =
+ hidden_rows.use_page_cache &&
!config::disable_storage_page_cache,
+ .file_reader = segment->file_reader().get(),
+ .stats = &stats,
+ .io_ctx = read_options.io_ctx,
+ };
+ RETURN_IF_ERROR(iterator->init(iterator_options));
+ RETURN_IF_ERROR(iterator->seek_to_ordinal(0));
+ auto data_type =
DataTypeFactory::instance().create_data_type(delete_sign);
+ constexpr size_t kBatch = 4096;
+ for (uint32_t begin = 0; begin < segment->num_rows();) {
+ size_t rows = std::min<size_t>(kBatch, segment->num_rows() -
begin);
+ MutableColumnPtr column = data_type->create_column();
+ bool has_null = false;
+ RETURN_IF_ERROR(iterator->next_batch(&rows, column,
&has_null));
+ if (rows == 0) {
+ return Status::Corruption("segment {} ended after {} of
its {} rows",
+ segment->id(), begin,
segment->num_rows());
+ }
+ const auto& signs = assert_cast<const
ColumnInt8&>(*column).get_data();
+ for (size_t i = 0; i < rows; ++i) {
+ if (signs[i] != 0) {
+ hidden.add(begin + static_cast<uint32_t>(i));
+ }
+ }
+ begin += static_cast<uint32_t>(rows);
+ }
+ }
+ }
+
+ hidden_docs->resize(hidden.cardinality());
+ hidden.toUint32Array(hidden_docs->data());
+ return Status::OK();
+}
+
+} // namespace
+
Status CollectionStatistics::collect(RuntimeState* state,
const std::vector<RowSetSplits>&
rs_splits,
const TabletSchemaSPtr& tablet_schema,
const VExprContextSPtrs&
common_expr_ctxs_push_down,
- io::IOContext* io_ctx) {
+ io::IOContext* io_ctx, const
MowHiddenRows* hidden_rows) {
std::vector<RowsetSharedPtr> rowsets;
rowsets.reserve(rs_splits.size());
for (const auto& rs_split : rs_splits) {
@@ -105,14 +295,14 @@ Status CollectionStatistics::collect(RuntimeState* state,
rowsets.emplace_back(std::move(rowset));
}
return collect_full_collection(state, rowsets, tablet_schema,
common_expr_ctxs_push_down,
- io_ctx);
+ io_ctx, hidden_rows);
}
// NOLINTNEXTLINE(readability-function-cognitive-complexity) Statistics
collection spans rowsets and formats.
Status CollectionStatistics::collect_full_collection(
RuntimeState* state, const std::vector<RowsetSharedPtr>& rowsets,
const TabletSchemaSPtr& tablet_schema, const VExprContextSPtrs&
common_expr_ctxs_push_down,
- io::IOContext* io_ctx) {
+ io::IOContext* io_ctx, const MowHiddenRows* hidden_rows) {
clear();
std::unordered_map<std::wstring, CollectInfo> collect_infos;
RETURN_IF_ERROR(
@@ -125,7 +315,8 @@ Status CollectionStatistics::collect_full_collection(
for (const auto& rowset : rowsets) {
DORIS_CHECK(rowset != nullptr);
for (auto seg : rowset->segments()) {
- auto status = process_segment(rowset, seg, tablet_schema.get(),
collect_infos, io_ctx);
+ auto status = process_segment(rowset, seg, tablet_schema.get(),
collect_infos,
+ hidden_rows, io_ctx);
if (!status.ok()) {
if (tablet_schema->get_inverted_index_storage_format() !=
InvertedIndexStorageFormatPB::SNII &&
@@ -242,6 +433,7 @@ Status CollectionStatistics::process_segment(const
RowsetSharedPtr& rowset,
const RowsetSegmentView& seg,
const TabletSchema* tablet_schema,
const CollectInfoMap&
collect_infos,
+ const MowHiddenRows* hidden_rows,
io::IOContext* io_ctx) {
auto seg_path = DORIS_TRY(seg.path());
auto rowset_meta = rowset->rowset_meta();
@@ -264,6 +456,14 @@ Status CollectionStatistics::process_segment(const
RowsetSharedPtr& rowset,
return init_status;
}
+ // Row ID lists are as large as the segment's hidden or NULL rows, so they
are charged to the
+ // query's memory.
+ DorisVector<uint32_t> hidden_docs;
+ if (hidden_rows != nullptr) {
+ RETURN_IF_ERROR(collect_hidden_docs(rowset, seg, *tablet_schema,
*hidden_rows, io_ctx,
+ &hidden_docs));
+ }
+
if (is_snii) {
segment_v2::snii_doris::DorisSniiFileReader::ScopedIOContext
io_context_scope(io_ctx);
SniiScoringSegmentAccumulator segment_accumulator;
@@ -285,6 +485,25 @@ Status CollectionStatistics::process_segment(const
RowsetSharedPtr& rowset,
::doris::snii::reader::SniiIndexSource source(*logical_reader);
index_query::IndexScoringStats scoring_stats;
RETURN_IF_ERROR(source.scoring_stats(&scoring_stats));
+ if (hidden_rows != nullptr) {
+ roaring::Roaring null_docs;
+ uint64_t non_null_length_sum = 0;
+ if (scoring_stats.has_norms) {
+ std::vector<uint32_t> null_doc_ids;
+
RETURN_IF_ERROR(logical_reader->read_null_docids(&null_doc_ids));
+ // The SNII reader returns a plain vector; charge it while
it lives.
+ const auto null_doc_id_bytes =
+ static_cast<int64_t>(null_doc_ids.capacity() *
sizeof(uint32_t));
+ CONSUME_THREAD_MEM_TRACKER(null_doc_id_bytes);
+ Defer release_null_doc_ids(
+ [null_doc_id_bytes] {
RELEASE_THREAD_MEM_TRACKER(null_doc_id_bytes); });
+ null_docs.addMany(null_doc_ids.size(),
null_doc_ids.data());
+
RETURN_IF_ERROR(collection_statistics_detail::sum_non_null_norm_lengths(
+ source, null_doc_ids, &non_null_length_sum));
+ }
+
RETURN_IF_ERROR(collection_statistics_detail::exclude_hidden_docs(
+ source, hidden_docs, null_docs, non_null_length_sum,
&scoring_stats));
+ }
const uint64_t segment_doc_count = scoring_stats.doc_count;
RETURN_IF_ERROR(admit_snii_scoring_segment(
ws_field_name, segment_doc_count,
scoring_stats.token_count,
@@ -299,6 +518,10 @@ Status CollectionStatistics::process_segment(const
RowsetSharedPtr& rowset,
segment_v2::inverted_index::StringHelper::to_wstring(logical_term_bytes);
uint64_t doc_frequency = 0;
RETURN_IF_ERROR(source.scoring_doc_freq(logical_term_bytes,
&doc_frequency));
+ if (!hidden_docs.empty()) {
+
RETURN_IF_ERROR(collection_statistics_detail::exclude_hidden_term_docs(
+ source, logical_term_bytes, hidden_docs,
&doc_frequency));
+ }
if (doc_frequency > segment_doc_count) {
return
Status::Error<ErrorCode::INVERTED_INDEX_FILE_CORRUPTED>(
"SNII term document frequency {} exceeds segment
document count {}",
@@ -311,6 +534,7 @@ Status CollectionStatistics::process_segment(const
RowsetSharedPtr& rowset,
}
commit_snii_scoring_segment(std::move(segment_accumulator));
+ _counted_hidden_docs = _counted_hidden_docs || !hidden_docs.empty();
return Status::OK();
}
@@ -361,6 +585,47 @@ Status CollectionStatistics::process_segment(const
RowsetSharedPtr& rowset,
ws_field_name, io_ctx);
index_query::IndexScoringStats scoring_stats;
RETURN_IF_ERROR(source->scoring_stats(&scoring_stats));
+ if (hidden_rows != nullptr) {
+ std::shared_ptr<roaring::Roaring> null_docs =
std::make_shared<roaring::Roaring>();
+ uint64_t non_null_length_sum = 0;
+ if (scoring_stats.has_norms) {
+ segment_v2::InvertedIndexQueryCacheHandle
null_bitmap_cache_handle;
+ RETURN_IF_ERROR(segment_v2::read_clucene_null_bitmap(
+ *idx_file_reader, *collect_info.index_meta, io_ctx,
+ &null_bitmap_cache_handle));
+ null_docs = null_bitmap_cache_handle.get_bitmap();
+ DORIS_CHECK(null_docs != nullptr);
+ auto sum_lengths = [&source, &null_docs](uint64_t* out) {
+ DorisVector<uint32_t>
null_doc_ids(null_docs->cardinality());
+ null_docs->toUint32Array(null_doc_ids.data());
+ return
collection_statistics_detail::sum_non_null_norm_lengths(
+ *source, null_doc_ids, out);
+ };
+#ifdef BE_TEST
+ RETURN_IF_ERROR(sum_lengths(&non_null_length_sum));
+#else
+
RETURN_IF_ERROR(inverted_index_cache_handle.get<CluceneSearcherCacheValue>()
+
->non_null_norm_length_sum(ws_field_name, sum_lengths,
+
&non_null_length_sum));
+#endif
+ }
+ // The statistics keep no norms, so the lengths of a segment's
hidden rows are summed
+ // once per set of hidden rows and kept with the searcher.
+ auto sum_hidden_lengths = [&](std::span<const uint32_t> docs,
uint64_t* out) {
+ auto sum = [&source, docs](uint64_t* lengths) {
+ return source->sum_norm_lengths_of(docs, lengths);
+ };
+#ifdef BE_TEST
+ return sum(out);
+#else
+ return
inverted_index_cache_handle.get<CluceneSearcherCacheValue>()
+ ->hidden_norm_length_sum(ws_field_name, docs, sum,
out);
+#endif
+ };
+ RETURN_IF_ERROR(collection_statistics_detail::exclude_hidden_docs(
+ hidden_docs, *null_docs, non_null_length_sum,
sum_hidden_lengths,
+ &scoring_stats));
+ }
total_segment_docs =
std::max(total_segment_docs,
static_cast<int32_t>(scoring_stats.doc_count));
// BM25 on an analyzed index needs the record length of every row, and
CLucene keeps
@@ -387,12 +652,17 @@ Status CollectionStatistics::process_segment(const
RowsetSharedPtr& rowset,
segment_v2::inverted_index::StringHelper::to_wstring(logical_term_bytes);
uint64_t doc_frequency = 0;
RETURN_IF_ERROR(source->scoring_doc_freq(logical_term_bytes,
&doc_frequency));
+ if (!hidden_docs.empty()) {
+
RETURN_IF_ERROR(collection_statistics_detail::exclude_hidden_term_docs(
+ *source, logical_term_bytes, hidden_docs,
&doc_frequency));
+ }
collection_statistics_detail::add_term_doc_frequency(&_term_doc_freqs,
ws_field_name,
logical_term,
doc_frequency);
}
}
_total_num_docs += static_cast<uint64_t>(total_segment_docs);
+ _counted_hidden_docs = _counted_hidden_docs || !hidden_docs.empty();
_avg_dl_by_col.clear();
_idf_by_col_term.clear();
@@ -440,6 +710,7 @@ void CollectionStatistics::commit_snii_scoring_segment(
void CollectionStatistics::clear() {
_total_num_docs = 0;
+ _counted_hidden_docs = false;
_total_num_tokens.clear();
_term_doc_freqs.clear();
_avg_dl_by_col.clear();
@@ -448,6 +719,7 @@ void CollectionStatistics::clear() {
void CollectionStatistics::copy_collected_from(const CollectionStatistics&
other) {
_total_num_docs = other._total_num_docs;
+ _counted_hidden_docs = other._counted_hidden_docs;
_total_num_tokens = other._total_num_tokens;
_term_doc_freqs = other._term_doc_freqs;
_avg_dl_by_col.clear();
@@ -457,13 +729,14 @@ void CollectionStatistics::copy_collected_from(const
CollectionStatistics& other
Status TabletCollectionStatistics::copy_to(RuntimeState* state,
const TabletSchemaSPtr&
tablet_schema,
const VExprContextSPtrs&
common_expr_ctxs_push_down,
- io::IOContext* io_ctx,
+ io::IOContext* io_ctx, const
MowHiddenRows* hidden_rows,
CollectionStatistics* statistics) {
DORIS_CHECK(statistics != nullptr);
std::lock_guard lock(_mutex);
if (!_collected) {
- _status = _statistics.collect_full_collection(state, _rowsets,
tablet_schema,
-
common_expr_ctxs_push_down, io_ctx);
+ // Every scanner of the tablet reads with the same delete bitmap and
version.
+ _status = _statistics.collect_full_collection(
+ state, _rowsets, tablet_schema, common_expr_ctxs_push_down,
io_ctx, hidden_rows);
_collected = true;
// Do not hold the rowsets any longer than needed.
_rowsets.clear();
@@ -504,7 +777,9 @@ uint64_t
CollectionStatistics::get_total_term_cnt_by_col(const std::wstring& luc
}
uint64_t CollectionStatistics::get_doc_num() const {
- if (_total_num_docs == 0) {
+ // A tablet whose rows are all hidden has no document a query can match,
so its statistics
+ // are empty rather than missing.
+ if (_total_num_docs == 0 && !_counted_hidden_docs) {
throw Exception(
ErrorCode::INVERTED_INDEX_CLUCENE_ERROR,
"Index statistics collection failed: No data available for
SimilarityCollector");
@@ -521,7 +796,7 @@ float CollectionStatistics::get_or_calculate_avg_dl(const
std::wstring& lucene_c
const uint64_t total_term_cnt = get_total_term_cnt_by_col(lucene_col_name);
const uint64_t total_doc_cnt = get_doc_num();
- float avg_dl = total_doc_cnt > 0 ? float((double)total_term_cnt /
(double)total_doc_cnt) : 0.0F;
+ float avg_dl = total_doc_cnt > 0 ? float((double)total_term_cnt /
(double)total_doc_cnt) : 1.0F;
_avg_dl_by_col[lucene_col_name] = avg_dl;
return avg_dl;
}
diff --git a/be/src/storage/index/inverted/reader/collection_statistics.h
b/be/src/storage/index/inverted/reader/collection_statistics.h
index 0da3daa2dd3..ebd52ce91e4 100644
--- a/be/src/storage/index/inverted/reader/collection_statistics.h
+++ b/be/src/storage/index/inverted/reader/collection_statistics.h
@@ -21,6 +21,7 @@
#include <functional>
#include <mutex>
#include <optional>
+#include <span>
#include <string>
#include <string_view>
#include <unordered_map>
@@ -34,8 +35,17 @@
#include "storage/index/inverted/reader/predicate_collector.h"
#include "storage/olap_common.h"
+namespace roaring {
+class Roaring;
+} // namespace roaring
+
namespace doris {
+namespace index_query {
+class IndexSource;
+struct IndexScoringStats;
+} // namespace index_query
+
namespace io {
class FileSystem;
using FileSystemSPtr = std::shared_ptr<FileSystem>;
@@ -44,6 +54,8 @@ struct IOContext;
struct RowSetSplits;
+class DeleteBitmap;
+
class Rowset;
class RowsetSegmentView;
using RowsetSharedPtr = std::shared_ptr<Rowset>;
@@ -52,19 +64,43 @@ class TabletIndex;
class TabletSchema;
using TabletSchemaSPtr = std::shared_ptr<TabletSchema>;
+// The rows of a merge-on-write tablet that its segments still hold but a scan
at `version` does
+// not return: rows a later load overwrote or deleted, which the delete bitmap
marks, and rows
+// written with the delete sign, which queries filter out. Compaction removes
both, on each
+// replica at its own time, so statistics counting them would differ between
the replicas of a
+// tablet, and so would every BM25 score. Rows written with the delete sign
are left out even when
+// the query returns them (`skip_delete_sign`, `show_hidden_columns`), and
rows deleted by a delete
+// predicate (`enable_mow_light_delete`) are still counted, as the delete
bitmap does not mark them.
+//
+// The cache flags carry the query's choices to the segment and column reads
of the delete sign;
+// the file cache choices come with the I/O context.
+struct MowHiddenRows {
+ std::shared_ptr<DeleteBitmap> delete_bitmap;
+ int64_t version = -1;
+ bool use_segment_cache = true;
+ bool use_page_cache = true;
+};
+
class CollectionStatistics {
public:
explicit CollectionStatistics(PredicateCollectorPtr search_collector =
nullptr)
: _search_collector(std::move(search_collector)) {}
virtual ~CollectionStatistics() = default;
+ // With `hidden_rows`, the statistics describe only the rows a
merge-on-write scan returns,
+ // so every replica of a tablet scores alike whatever it has compacted:
the hidden rows are
+ // left out of the document count and the term document frequencies, and
the token count is
+ // the sum of the document lengths the norms of the returned non-NULL rows
stand for, or the
+ // returned row count for an index without norms.
Status collect(RuntimeState* state, const std::vector<RowSetSplits>&
rs_splits,
const TabletSchemaSPtr& tablet_schema,
- const VExprContextSPtrs& common_expr_ctxs_push_down,
io::IOContext* io_ctx);
+ const VExprContextSPtrs& common_expr_ctxs_push_down,
io::IOContext* io_ctx,
+ const MowHiddenRows* hidden_rows = nullptr);
Status collect_full_collection(RuntimeState* state, const
std::vector<RowsetSharedPtr>& rowsets,
const TabletSchemaSPtr& tablet_schema,
const VExprContextSPtrs&
common_expr_ctxs_push_down,
- io::IOContext* io_ctx);
+ io::IOContext* io_ctx,
+ const MowHiddenRows* hidden_rows = nullptr);
MOCK_FUNCTION float get_or_calculate_idf(const std::wstring&
lucene_col_name,
const std::wstring& term);
@@ -87,7 +123,7 @@ private:
CollectInfoMap* collect_infos);
Status process_segment(const RowsetSharedPtr& rowset, const
RowsetSegmentView& seg,
const TabletSchema* tablet_schema, const
CollectInfoMap& collect_infos,
- io::IOContext* io_ctx);
+ const MowHiddenRows* hidden_rows, io::IOContext*
io_ctx);
Status admit_snii_scoring_segment(const std::wstring& field_name, uint64_t
index_doc_count,
uint64_t sum_total_term_freq, bool
has_positions,
bool has_norms,
@@ -102,6 +138,9 @@ private:
uint64_t get_doc_num() const;
uint64_t _total_num_docs = 0;
+ // Whether a collected segment held rows the scan does not return, so that
no documents can
+ // mean every row is hidden.
+ bool _counted_hidden_docs = false;
std::unordered_map<std::wstring, uint64_t> _total_num_tokens;
std::unordered_map<std::wstring, std::unordered_map<std::wstring,
uint64_t>> _term_doc_freqs;
@@ -133,7 +172,7 @@ public:
// Later callers wait for the first collection and receive its result or
error.
Status copy_to(RuntimeState* state, const TabletSchemaSPtr& tablet_schema,
const VExprContextSPtrs& common_expr_ctxs_push_down,
io::IOContext* io_ctx,
- CollectionStatistics* statistics);
+ const MowHiddenRows* hidden_rows, CollectionStatistics*
statistics);
private:
std::mutex _mutex;
@@ -167,6 +206,34 @@ void add_term_doc_frequency(
logical_frequencies,
const std::wstring& field, const std::wstring& logical_term, uint64_t
doc_frequency);
+// The sum of the lengths the norms of a segment's documents other than the
ascending `null_docs`
+// stand for. A NULL row is a document without tokens, though its norm stands
for one. Rows that are
+// indexed without tokens but are not NULL keep a norm that cannot be told
from a one-token row's:
+// an analyzed empty string on both formats, and on SNII any text that yields
no token.
+Status sum_non_null_norm_lengths(index_query::IndexSource& source,
+ std::span<const uint32_t> null_docs,
uint64_t* out);
+
+// Turns a segment's scoring statistics into those of its documents other than
the ascending
+// `hidden_docs`: their count, and the sum of the lengths the norms of the
non-NULL ones stand
+// for, or their count when the index has no norms. `non_null_length_sum` is
the segment's
+// sum_non_null_norm_lengths(), and `null_docs` its NULL rows; both are unused
without norms.
+Status exclude_hidden_docs(index_query::IndexSource& source, std::span<const
uint32_t> hidden_docs,
+ const roaring::Roaring& null_docs, uint64_t
non_null_length_sum,
+ index_query::IndexScoringStats* stats);
+
+// Sums the lengths the norms of ascending rows of a segment's index stand for.
+using SumNormLengthsOf = std::function<Status(std::span<const uint32_t> docs,
uint64_t* out)>;
+
+// exclude_hidden_docs, the lengths of the hidden non-NULL rows summed by
`sum_norm_lengths_of`.
+Status exclude_hidden_docs(std::span<const uint32_t> hidden_docs, const
roaring::Roaring& null_docs,
+ uint64_t non_null_length_sum,
+ const SumNormLengthsOf& sum_norm_lengths_of,
+ index_query::IndexScoringStats* stats);
+
+// Takes from a term's document frequency the documents of `hidden_docs` that
hold it.
+Status exclude_hidden_term_docs(index_query::IndexSource& source,
std::string_view term,
+ std::span<const uint32_t> hidden_docs,
uint64_t* doc_frequency);
+
} // namespace collection_statistics_detail
} // namespace doris
diff --git a/be/src/storage/index/inverted/snii/reader/snii_index_source.cpp
b/be/src/storage/index/inverted/snii/reader/snii_index_source.cpp
index 95b0d2cd023..ef219a734a2 100644
--- a/be/src/storage/index/inverted/snii/reader/snii_index_source.cpp
+++ b/be/src/storage/index/inverted/snii/reader/snii_index_source.cpp
@@ -217,6 +217,24 @@ Status SniiIndexSource::scoring_doc_freq(std::string_view
term, uint64_t* out) {
return Status::OK();
}
+// Reads the term as scoring_doc_freq does, so terms of a reserved namespace
resolve too.
+Status SniiIndexSource::scoring_doc_freq_among(std::string_view term,
+ std::span<const uint32_t> docs,
uint64_t* out) {
+ *out = 0;
+ if (docs.empty()) {
+ return Status::OK();
+ }
+ Term* resolved = nullptr;
+ RETURN_IF_ERROR(_resolve(term, &resolved, &_scoring_dict_cache));
+ if (!resolved->hit.found) {
+ return Status::OK();
+ }
+ std::unique_ptr<SniiPostingsCursor> cursor;
+ RETURN_IF_ERROR(
+ _cursor(*resolved, /*positions=*/false, /*scoring=*/false,
/*wave=*/nullptr, &cursor));
+ return index_query::count_listed_docs(*cursor, docs, out);
+}
+
// The norms section is read once, when a document's norm is first asked.
Status SniiIndexSource::encoded_norms(std::span<const uint32_t> docs,
std::vector<uint32_t>* out) {
out->assign(docs.size(), 1);
diff --git a/be/src/storage/index/inverted/snii/reader/snii_index_source.h
b/be/src/storage/index/inverted/snii/reader/snii_index_source.h
index 0546d7c1969..ef46af713f1 100644
--- a/be/src/storage/index/inverted/snii/reader/snii_index_source.h
+++ b/be/src/storage/index/inverted/snii/reader/snii_index_source.h
@@ -57,6 +57,8 @@ public:
Status may_hold(std::string_view term, bool* held) override;
Status doc_freq(std::string_view term, uint64_t* out) override;
Status scoring_doc_freq(std::string_view term, uint64_t* out) override;
+ Status scoring_doc_freq_among(std::string_view term, std::span<const
uint32_t> docs,
+ uint64_t* out) override;
Status encoded_norms(std::span<const uint32_t> docs,
std::vector<uint32_t>* out) override;
Status expand_terms(index_query::TermPattern& pattern, int32_t
max_expansions,
std::vector<std::string>* out) override;
diff --git a/be/test/exec/scan/olap_scanner_test.cpp
b/be/test/exec/scan/olap_scanner_test.cpp
index a3f20ef6b57..51436d9ce95 100644
--- a/be/test/exec/scan/olap_scanner_test.cpp
+++ b/be/test/exec/scan/olap_scanner_test.cpp
@@ -102,6 +102,23 @@ TEST(OlapScannerTest,
BuildScoreRuntimeCollectionIoContextAllowsMissingQueryCont
EXPECT_EQ(io_ctx.file_cache_stats, &stats);
EXPECT_TRUE(io_ctx.is_inverted_index);
EXPECT_EQ(io_ctx.remote_scan_cache_write_limiter, nullptr);
+ EXPECT_FALSE(io_ctx.is_disposable);
+ EXPECT_FALSE(io_ctx.inverted_index_snii_read_no_write_file_cache);
+}
+
+// The statistics read the index before the scan, with the query's file cache
choices.
+TEST(OlapScannerTest,
BuildScoreRuntimeCollectionIoContextFollowsQueryFileCacheOptions) {
+ MockRuntimeState state;
+ state._query_id = make_query_id();
+ state._query_ctx = nullptr;
+ state._query_options.__set_disable_file_cache(true);
+
state._query_options.__set_inverted_index_snii_read_no_write_file_cache(true);
+
+ io::FileCacheStatistics stats;
+ auto io_ctx = build_score_runtime_collection_io_context(&state,
ReaderType::READER_QUERY, 3600,
+ &stats);
+ EXPECT_TRUE(io_ctx.is_disposable);
+ EXPECT_TRUE(io_ctx.inverted_index_snii_read_no_write_file_cache);
}
} // namespace doris
diff --git
a/be/test/storage/index/inverted/query/engine/multi_segment_collector_test.cpp
b/be/test/storage/index/inverted/query/engine/multi_segment_collector_test.cpp
index 74affbaa8be..67975b91521 100644
---
a/be/test/storage/index/inverted/query/engine/multi_segment_collector_test.cpp
+++
b/be/test/storage/index/inverted/query/engine/multi_segment_collector_test.cpp
@@ -17,6 +17,23 @@
#include <CLucene.h>
#include <CLucene/index/MultiReader.h>
+#ifdef __clang__
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wshadow-field"
+#pragma clang diagnostic ignored "-Woverloaded-virtual"
+#pragma clang diagnostic ignored "-Winconsistent-missing-override"
+#pragma clang diagnostic ignored "-Wreorder-ctor"
+#pragma clang diagnostic ignored "-Wshorten-64-to-32"
+#elif defined(__GNUC__)
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Woverloaded-virtual"
+#endif
+#include <CLucene/index/_MultiSegmentReader.h>
+#ifdef __clang__
+#pragma clang diagnostic pop
+#elif defined(__GNUC__)
+#pragma GCC diagnostic pop
+#endif
#include <gtest/gtest.h>
#include <functional>
@@ -44,6 +61,7 @@
#include "storage/index/inverted/query/engine/phrase_query/phrase_query.h"
#include "storage/index/inverted/query/engine/term_query/term_query.h"
#include "storage/index/inverted/query/exec/docid_sink.h"
+#include "storage/index/inverted/query/scoring/bm25_similarity.h"
#include "storage/index/inverted/query/scoring/collection_similarity.h"
#include "storage/index/inverted/reader/collection_statistics.h"
#include "storage/index/inverted/snii/reader/snii_index_source.h"
@@ -81,7 +99,7 @@ public:
EXPECT_TRUE(io::global_local_filesystem()->delete_directory(kTestDir).ok());
}
-private:
+protected:
static void create_test_index(const std::string& dir, const
std::vector<std::string>& docs,
int32_t max_buffered_docs = 100,
const std::string& field_name = "title",
@@ -875,4 +893,44 @@ TEST_F(MultiSegmentCollectorTest,
CollectDocSetWithSingleReaderBinding) {
_CLDECDELETE(dir1);
}
+// An index of several CLucene segments opens as a reader of several segments.
Its norm length
+// sums are read per segment and equal those of the reader's combined norms.
+TEST_F(MultiSegmentCollectorTest, NormLengthSumsReadEachSegment) {
+ const std::string dir = kTestDir + "/segmented_norms";
+ ASSERT_TRUE(io::global_local_filesystem()->create_directory(dir).ok());
+ std::string long_doc;
+ for (int i = 0; i < 30; ++i) {
+ long_doc += "word ";
+ }
+ create_test_index(dir, {"a b", "c", long_doc, "d e f", "g", "h i"}, 2,
"title", true);
+
+ auto reader =
make_shared_reader(lucene::index::IndexReader::open(dir.c_str()));
+ auto* multi_segment =
dynamic_cast<lucene::index::MultiSegmentReader*>(reader.get());
+ ASSERT_NE(multi_segment, nullptr);
+ ASSERT_EQ(multi_segment->getSubReaders()->length, 3);
+ auto source = clucene_index_source(reader, L"title", nullptr);
+ uint64_t all = 0;
+ ASSERT_TRUE(source->sum_norm_lengths(&all).ok());
+ const std::vector<uint32_t> docs {0, 2, 3, 5};
+ uint64_t some = 0;
+ ASSERT_TRUE(source->sum_norm_lengths_of(docs, &some).ok());
+
+ auto combined =
make_shared_reader(lucene::index::IndexReader::open(dir.c_str()));
+ const uint8_t* norms = combined->norms(L"title");
+ ASSERT_NE(norms, nullptr);
+ const auto lengths = BM25Similarity::lucene_norm_lengths();
+ uint64_t expected_all = 0;
+ for (int32_t doc = 0; doc < combined->maxDoc(); ++doc) {
+ expected_all += static_cast<uint64_t>(lengths[norms[doc]]);
+ }
+ uint64_t expected_some = 0;
+ for (const uint32_t doc : docs) {
+ expected_some += static_cast<uint64_t>(lengths[norms[doc]]);
+ }
+ EXPECT_EQ(all, expected_all);
+ EXPECT_EQ(some, expected_some);
+ // Lengths below 24 are exact; the 30-token row reads back approximately.
+ EXPECT_EQ(expected_all - static_cast<uint64_t>(lengths[norms[2]]), 9);
+}
+
} // namespace doris::segment_v2
diff --git
a/be/test/storage/index/inverted/reader/collection_statistics_test.cpp
b/be/test/storage/index/inverted/reader/collection_statistics_test.cpp
index 46a17c39c8c..a6b52573770 100644
--- a/be/test/storage/index/inverted/reader/collection_statistics_test.cpp
+++ b/be/test/storage/index/inverted/reader/collection_statistics_test.cpp
@@ -23,10 +23,14 @@
#include <atomic>
#include <barrier>
+#include <cmath>
#include <future>
#include <memory>
+#include <roaring/roaring.hh>
+#include <set>
#include <string>
#include <thread>
+#include <variant>
#include <vector>
#include "common/exception.h"
@@ -47,8 +51,10 @@
#include "storage/index/index_file_writer.h"
#include "storage/index/index_writer.h"
#include "storage/index/inverted/analysis/analyzer/analyzer.h"
+#include "storage/index/inverted/common/inverted_index_cache.h"
#include "storage/index/inverted/common/inverted_index_desc.h"
#include "storage/index/inverted/common/string_helper.h"
+#include "storage/index/inverted/query/spi/fake_index_source.h"
#include "storage/index/inverted/reader/predicate_collector.h"
#include "storage/index/inverted/snii/format/phrase_bigram.h"
#include "storage/index/inverted/snii/query/bm25_scorer.h"
@@ -59,6 +65,7 @@
#include "storage/rowset/rowset.h"
#include "storage/rowset/rowset_meta.h"
#include "storage/rowset/rowset_reader.h"
+#include "storage/tablet/tablet_meta.h"
#include "storage/tablet/tablet_schema.h"
#include "testutil/mock/mock_runtime_state.h"
#include "util/defer_op.h"
@@ -67,6 +74,9 @@
namespace doris {
using collection_statistics_detail::add_term_doc_frequency;
+using collection_statistics_detail::exclude_hidden_docs;
+using collection_statistics_detail::exclude_hidden_term_docs;
+using collection_statistics_detail::sum_non_null_norm_lengths;
using collection_statistics_detail::resolve_snii_scoring_segment;
using collection_statistics_detail::SniiScoringSegmentStats;
@@ -322,11 +332,18 @@ protected:
test_dir_ = "./collection_statistics_test_" +
std::to_string(::testing::UnitTest::GetInstance()->random_seed());
ASSERT_TRUE(io::global_local_filesystem()->create_directory(test_dir_).ok());
+ // Merge-on-write collection reads the null bitmap through the query
cache.
+ previous_query_cache_ =
ExecEnv::GetInstance()->get_inverted_index_query_cache();
+ query_cache_.reset(
+ segment_v2::InvertedIndexQueryCache::create_global_cache(1024
* 1024, 1));
+
ExecEnv::GetInstance()->set_inverted_index_query_cache(query_cache_.get());
}
void TearDown() override {
stats_.reset();
runtime_state_.reset();
+
ExecEnv::GetInstance()->set_inverted_index_query_cache(previous_query_cache_);
+ query_cache_.reset();
(void)io::global_local_filesystem()->delete_directory(test_dir_);
}
@@ -441,8 +458,11 @@ protected:
return tablet_schema;
}
+ // Writes `rows`, the rows of `null_rows` as NULL whatever their value.
Status write_legacy_v3_segment(const TabletSchemaSPtr& tablet_schema,
- const std::string& segment_path, size_t
column_ordinal = 0) {
+ const std::string& segment_path, size_t
column_ordinal = 0,
+ const std::vector<std::string>& rows =
{"alpha beta"},
+ const std::set<size_t>& null_rows = {}) {
std::vector<StorePath> paths;
paths.emplace_back(test_dir_, 1024);
auto tmp_file_dirs = std::make_unique<segment_v2::TmpFileDirs>(paths);
@@ -467,9 +487,15 @@ protected:
std::unique_ptr<segment_v2::IndexColumnWriter> column_writer;
RETURN_IF_ERROR(segment_v2::IndexColumnWriter::create(&column,
&column_writer, &file_writer,
index_metas[0]));
- std::vector<Slice> values {Slice("alpha beta")};
- RETURN_IF_ERROR(column_writer->add_values(reinterpret_cast<const
uint8_t*>(values.data()),
- values.size()));
+ for (size_t row = 0; row < rows.size(); ++row) {
+ if (null_rows.contains(row)) {
+ RETURN_IF_ERROR(column_writer->add_nulls(1));
+ } else {
+ const Slice value(rows[row]);
+ RETURN_IF_ERROR(
+ column_writer->add_values(reinterpret_cast<const
uint8_t*>(&value), 1));
+ }
+ }
RETURN_IF_ERROR(column_writer->finish());
RETURN_IF_ERROR(file_writer.begin_close());
return file_writer.finish_close();
@@ -723,6 +749,8 @@ protected:
std::unique_ptr<CollectionStatistics> stats_;
std::shared_ptr<MockRuntimeState> runtime_state_;
std::string test_dir_;
+ segment_v2::InvertedIndexQueryCache* previous_query_cache_ = nullptr;
+ std::unique_ptr<segment_v2::InvertedIndexQueryCache> query_cache_;
};
TEST_F(CollectionStatisticsTest, CollectWithEmptyRowsetSplits) {
@@ -931,6 +959,384 @@ TEST_F(CollectionStatisticsTest,
MissingFieldIndexPreservesOtherFieldStatistics)
expect_collected_term(L"2", L"alpha", 1);
}
+// A replica whose segment still holds a row that a later load overwrote must
collect the same
+// statistics as a replica that compacted the row away, or the two score every
row differently.
+TEST_F(CollectionStatisticsTest, HiddenRowsAreCollectedLikeCompactedAway) {
+ auto tablet_schema = create_legacy_v3_schema();
+ // Forty-five tokens: CLucene's norm holds lengths exactly only up to 40.
+ std::string overwritten;
+ for (int i = 0; i < 44; ++i) {
+ overwritten += "gamma ";
+ }
+ overwritten += "alpha";
+ const std::string uncompacted_path = test_dir_ +
"/hidden_rows_uncompacted_0.dat";
+ const std::string compacted_path = test_dir_ +
"/hidden_rows_compacted_0.dat";
+ ASSERT_TRUE(write_legacy_v3_segment(tablet_schema, uncompacted_path, 0,
+ {"alpha beta", overwritten, "beta
gamma delta"})
+ .ok());
+ ASSERT_TRUE(write_legacy_v3_segment(tablet_schema, compacted_path, 0,
+ {"alpha beta", "beta gamma delta"})
+ .ok());
+
+ struct Collected {
+ uint64_t doc_count;
+ uint64_t token_count;
+ uint64_t alpha_doc_frequency;
+ uint64_t gamma_doc_frequency;
+ float avg_dl;
+ float alpha_idf;
+ };
+ // The delete bitmap's aggregation cache is keyed by tablet and rowset,
and the mock rowsets
+ // share their ids, so each collection uses its own tablet id.
+ int64_t next_tablet_id = 9100;
+ auto collect = [&](const std::string& segment_path,
+ const std::vector<uint32_t>& hidden_rows) -> Collected {
+ auto rowset_meta =
std::make_shared<collection_statistics::MockRowsetMeta>();
+ auto rowset =
+
std::make_shared<collection_statistics::MockRowset>(tablet_schema, rowset_meta);
+ rowset->set_num_segments(1);
+ rowset->set_segment_path(0, segment_path);
+ auto reader =
std::make_shared<collection_statistics::MockRowsetReader>(rowset);
+ std::vector<RowSetSplits> splits {RowSetSplits(reader)};
+
+ constexpr int64_t kVersion = 5;
+ MowHiddenRows hidden {.delete_bitmap =
std::make_shared<DeleteBitmap>(next_tablet_id++),
+ .version = kVersion};
+ for (const auto seg : rowset->segments()) {
+ for (const uint32_t row : hidden_rows) {
+ hidden.delete_bitmap->add(seg.delete_bitmap_key(kVersion),
row);
+ }
+ }
+ CollectionStatistics stats;
+ const Status status =
+ stats.collect(runtime_state_.get(), splits, tablet_schema,
+ create_match_expr_contexts("alpha gamma"),
nullptr, &hidden);
+ EXPECT_TRUE(status.ok()) << status;
+ return {.doc_count = stats.get_doc_num(),
+ .token_count = stats.get_total_term_cnt_by_col(L"1"),
+ .alpha_doc_frequency = stats.get_term_doc_freq_by_col(L"1",
L"alpha"),
+ .gamma_doc_frequency = stats.get_term_doc_freq_by_col(L"1",
L"gamma"),
+ .avg_dl = stats.get_or_calculate_avg_dl(L"1"),
+ .alpha_idf = stats.get_or_calculate_idf(L"1", L"alpha")};
+ };
+
+ const Collected uncompacted = collect(uncompacted_path, {1});
+ const Collected compacted = collect(compacted_path, {});
+ EXPECT_EQ(uncompacted.doc_count, 2);
+ EXPECT_EQ(uncompacted.token_count, 5);
+ EXPECT_EQ(uncompacted.alpha_doc_frequency, 1);
+ EXPECT_EQ(uncompacted.gamma_doc_frequency, 1);
+ EXPECT_EQ(uncompacted.doc_count, compacted.doc_count);
+ EXPECT_EQ(uncompacted.token_count, compacted.token_count);
+ EXPECT_EQ(uncompacted.alpha_doc_frequency, compacted.alpha_doc_frequency);
+ EXPECT_EQ(uncompacted.gamma_doc_frequency, compacted.gamma_doc_frequency);
+ EXPECT_EQ(uncompacted.avg_dl, compacted.avg_dl);
+ EXPECT_EQ(uncompacted.alpha_idf, compacted.alpha_idf);
+
+ // Without the delete bitmap the overwritten row is counted.
+ auto rowset_meta =
std::make_shared<collection_statistics::MockRowsetMeta>();
+ auto rowset =
std::make_shared<collection_statistics::MockRowset>(tablet_schema, rowset_meta);
+ rowset->set_num_segments(1);
+ rowset->set_segment_path(0, uncompacted_path);
+ auto reader =
std::make_shared<collection_statistics::MockRowsetReader>(rowset);
+ std::vector<RowSetSplits> splits {RowSetSplits(reader)};
+ ASSERT_TRUE(stats_->collect(runtime_state_.get(), splits, tablet_schema,
+ create_match_expr_contexts("alpha gamma"),
nullptr)
+ .ok());
+ expect_collected_stats(L"1", 3, 50);
+ expect_collected_term(L"1", L"alpha", 2);
+}
+
+// A tablet whose rows were all deleted or overwritten still has segments to
collect from, but no
+// document a query can match. Its statistics are empty, not missing.
+TEST_F(CollectionStatisticsTest, AllRowsHiddenCollectsEmptyStatistics) {
+ auto tablet_schema = create_legacy_v3_schema();
+ const std::string segment_path = test_dir_ + "/all_rows_hidden_0.dat";
+ ASSERT_TRUE(
+ write_legacy_v3_segment(tablet_schema, segment_path, 0, {"alpha
beta", "alpha"}).ok());
+
+ auto rowset_meta =
std::make_shared<collection_statistics::MockRowsetMeta>();
+ auto rowset =
std::make_shared<collection_statistics::MockRowset>(tablet_schema, rowset_meta);
+ rowset->set_num_segments(1);
+ rowset->set_segment_path(0, segment_path);
+ auto reader =
std::make_shared<collection_statistics::MockRowsetReader>(rowset);
+ std::vector<RowSetSplits> splits {RowSetSplits(reader)};
+
+ constexpr int64_t kVersion = 3;
+ MowHiddenRows hidden {.delete_bitmap =
std::make_shared<DeleteBitmap>(9200),
+ .version = kVersion};
+ for (const auto seg : rowset->segments()) {
+ hidden.delete_bitmap->add(seg.delete_bitmap_key(kVersion), 0);
+ hidden.delete_bitmap->add(seg.delete_bitmap_key(kVersion), 1);
+ }
+ const Status status = stats_->collect(runtime_state_.get(), splits,
tablet_schema,
+ create_match_expr_contexts("alpha"),
nullptr, &hidden);
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_EQ(stats_->get_doc_num(), 0);
+ expect_collected_term(L"1", L"alpha", 0);
+ EXPECT_TRUE(std::isfinite(stats_->get_or_calculate_avg_dl(L"1")));
+ EXPECT_TRUE(std::isfinite(stats_->get_or_calculate_idf(L"1", L"alpha")));
+
+ // Without any collected segment the statistics are still reported missing.
+ CollectionStatistics nothing_collected;
+ EXPECT_THROW(nothing_collected.get_doc_num(), Exception);
+}
+
+// NULL rows have no tokens. A merge-on-write tablet, whose token count comes
from the norms, counts
+// them as documents without tokens, as a table without a delete bitmap does.
+TEST_F(CollectionStatisticsTest, NullRowsCountAsDocumentsWithoutTokens) {
+ auto tablet_schema = create_legacy_v3_schema();
+ const std::string segment_path = test_dir_ + "/null_rows_0.dat";
+ ASSERT_TRUE(write_legacy_v3_segment(tablet_schema, segment_path, 0,
+ {"alpha beta", "", "alpha", "", "beta
gamma delta"}, {1, 3})
+ .ok());
+ auto collect = [&](const MowHiddenRows* hidden) {
+ auto rowset_meta =
std::make_shared<collection_statistics::MockRowsetMeta>();
+ auto rowset =
+
std::make_shared<collection_statistics::MockRowset>(tablet_schema, rowset_meta);
+ rowset->set_num_segments(1);
+ rowset->set_segment_path(0, segment_path);
+ auto reader =
std::make_shared<collection_statistics::MockRowsetReader>(rowset);
+ std::vector<RowSetSplits> splits {RowSetSplits(reader)};
+ CollectionStatistics stats;
+ const Status status = stats.collect(runtime_state_.get(), splits,
tablet_schema,
+
create_match_expr_contexts("alpha"), nullptr, hidden);
+ EXPECT_TRUE(status.ok()) << status;
+ return std::make_pair(stats.get_doc_num(),
stats.get_total_term_cnt_by_col(L"1"));
+ };
+
+ const auto without_bitmap = collect(nullptr);
+ EXPECT_EQ(without_bitmap.first, 5);
+ EXPECT_EQ(without_bitmap.second, 6);
+ MowHiddenRows nothing_hidden {.delete_bitmap =
std::make_shared<DeleteBitmap>(9300),
+ .version = 2};
+ EXPECT_EQ(collect(¬hing_hidden), without_bitmap);
+
+ // A hidden NULL row leaves the document count and still adds no tokens.
+ MowHiddenRows null_hidden {.delete_bitmap =
std::make_shared<DeleteBitmap>(9301), .version = 2};
+ auto rowset_meta =
std::make_shared<collection_statistics::MockRowsetMeta>();
+ auto rowset =
std::make_shared<collection_statistics::MockRowset>(tablet_schema, rowset_meta);
+ rowset->set_num_segments(1);
+ for (const auto seg : rowset->segments()) {
+ null_hidden.delete_bitmap->add(seg.delete_bitmap_key(2), 1);
+ }
+ EXPECT_EQ(collect(&null_hidden), std::make_pair(uint64_t {4}, uint64_t
{6}));
+}
+
+namespace {
+
+// A cursor listing its documents in the given blocks, a pair standing for a
dense range.
+class BlockListCursor final : public index_query::PostingsCursor {
+public:
+ using Block = std::variant<std::vector<uint32_t>, std::pair<uint32_t,
uint32_t>>;
+
+ explicit BlockListCursor(std::vector<Block> blocks) :
_blocks(std::move(blocks)) {}
+
+ uint32_t doc_freq() const override {
+ uint32_t docs = 0;
+ for (const Block& block : _blocks) {
+ if (const auto* listed =
std::get_if<std::vector<uint32_t>>(&block)) {
+ docs += static_cast<uint32_t>(listed->size());
+ } else {
+ const auto& [begin, end] = std::get<std::pair<uint32_t,
uint32_t>>(block);
+ docs += end - begin;
+ }
+ }
+ return docs;
+ }
+
+ Status next_block(index_query::PostingsBlock* block, bool* eof) override {
+ *block = {};
+ *eof = _next == _blocks.size();
+ if (*eof) {
+ return Status::OK();
+ }
+ const Block& next = _blocks[_next++];
+ if (const auto* docs = std::get_if<std::vector<uint32_t>>(&next)) {
+ block->docs = *docs;
+ } else {
+ const auto& [begin, end] = std::get<std::pair<uint32_t,
uint32_t>>(next);
+ block->dense = true;
+ block->range_begin = begin;
+ block->range_end = end;
+ }
+ return Status::OK();
+ }
+
+ Status seek_block(uint32_t target, index_query::PostingsBlock* block,
bool* eof) override {
+ ++seeks;
+ while (_next < _blocks.size() && last_doc(_blocks[_next]) < target) {
+ ++_next;
+ }
+ return next_block(block, eof);
+ }
+
+ Status shallow_seek(uint32_t /*target*/, bool* moved) override {
+ *moved = false;
+ return Status::OK();
+ }
+
+ index_query::BlockBound current_block_bound() const override { return {}; }
+
+ size_t seeks = 0;
+
+private:
+ static uint32_t last_doc(const Block& block) {
+ if (const auto* docs = std::get_if<std::vector<uint32_t>>(&block)) {
+ return docs->back();
+ }
+ return std::get<std::pair<uint32_t, uint32_t>>(block).second - 1;
+ }
+
+ std::vector<Block> _blocks;
+ size_t _next = 0;
+};
+
+} // namespace
+
+TEST(CollectionStatisticsHiddenRowsTest,
CountListedDocsScansSparseAndDenseBlocks) {
+ BlockListCursor cursor({std::vector<uint32_t> {1, 4, 6},
std::pair<uint32_t, uint32_t> {10, 20},
+ std::vector<uint32_t> {25, 30},
std::vector<uint32_t> {40, 41}});
+ const std::vector<uint32_t> docs {0, 4, 5, 12, 19, 20, 30, 31};
+ uint64_t listed = 0;
+ ASSERT_TRUE(index_query::count_listed_docs(cursor, docs, &listed).ok());
+ EXPECT_EQ(listed, 4); // 4, 12, 19 and 30.
+ // As many docs as postings: every block is decoded once, without seeking.
+ EXPECT_EQ(cursor.seeks, 0);
+}
+
+// Few docs among many postings: the cursor seeks to each doc instead of
decoding every block.
+TEST(CollectionStatisticsHiddenRowsTest, CountListedDocsLeapsOverFewDocs) {
+ BlockListCursor cursor(
+ {std::vector<uint32_t> {1, 4, 6}, std::pair<uint32_t, uint32_t>
{10, 5000},
+ std::vector<uint32_t> {6000, 7000}, std::vector<uint32_t> {8000,
8001}});
+ const std::vector<uint32_t> docs {4, 4999, 7500};
+ uint64_t listed = 0;
+ ASSERT_TRUE(index_query::count_listed_docs(cursor, docs, &listed).ok());
+ EXPECT_EQ(listed, 2); // 4 and 4999.
+ EXPECT_EQ(cursor.seeks, 3);
+}
+
+TEST(CollectionStatisticsHiddenRowsTest, CountListedDocsOfNothing) {
+ BlockListCursor cursor({std::vector<uint32_t> {1, 2}});
+ uint64_t listed = 7;
+ ASSERT_TRUE(index_query::count_listed_docs(cursor, {}, &listed).ok());
+ EXPECT_EQ(listed, 0);
+ BlockListCursor empty({});
+ const std::vector<uint32_t> docs {1, 2};
+ ASSERT_TRUE(index_query::count_listed_docs(empty, docs, &listed).ok());
+ EXPECT_EQ(listed, 0);
+ EXPECT_EQ(empty.seeks, 0);
+}
+
+// Scanning walks the wanted documents alongside the postings, however large
their IDs, and stops
+// once every wanted document is passed.
+TEST(CollectionStatisticsHiddenRowsTest,
CountListedDocsScansFarDocsWithoutWalkingEveryBlock) {
+ BlockListCursor cursor({std::vector<uint32_t> {3}, std::pair<uint32_t,
uint32_t> {10, 20},
+ std::vector<uint32_t> {9999999},
std::vector<uint32_t> {10000005}});
+ const std::vector<uint32_t> docs {3, 15, 19, 9999999};
+ uint64_t listed = 0;
+ ASSERT_TRUE(index_query::count_listed_docs(cursor, docs, &listed).ok());
+ EXPECT_EQ(listed, 4);
+ EXPECT_EQ(cursor.seeks, 0);
+
+ BlockListCursor far({std::vector<uint32_t> {1}});
+ const std::vector<uint32_t> far_docs {9999999};
+ ASSERT_TRUE(index_query::count_listed_docs(far, far_docs, &listed).ok());
+ EXPECT_EQ(listed, 0);
+}
+
+TEST(CollectionStatisticsHiddenRowsTest,
ExcludeHiddenDocsSumsNormsOfReturnedRows) {
+ index_query::testing::FakeIndexSource source;
+ source.set_doc_count(4);
+ source.norms = {{0, 3}, {1, 5}, {2, 2}, {3, 7}};
+ uint64_t non_null_length_sum = 0;
+ ASSERT_TRUE(sum_non_null_norm_lengths(source, {},
&non_null_length_sum).ok());
+ EXPECT_EQ(non_null_length_sum, 17);
+ const roaring::Roaring no_nulls;
+ index_query::IndexScoringStats stats {
+ .doc_count = 4, .token_count = 1000, .has_positions = true,
.has_norms = true};
+ const std::vector<uint32_t> hidden {1, 3};
+ ASSERT_TRUE(exclude_hidden_docs(source, hidden, no_nulls,
non_null_length_sum, &stats).ok());
+ EXPECT_EQ(stats.doc_count, 2);
+ EXPECT_EQ(stats.token_count, 5);
+
+ // With nothing hidden the token count still comes from the norms, as on a
replica that
+ // compacted the hidden rows away.
+ stats = {.doc_count = 4, .token_count = 1000, .has_positions = true,
.has_norms = true};
+ ASSERT_TRUE(exclude_hidden_docs(source, {}, no_nulls, non_null_length_sum,
&stats).ok());
+ EXPECT_EQ(stats.doc_count, 4);
+ EXPECT_EQ(stats.token_count, 17);
+}
+
+TEST(CollectionStatisticsHiddenRowsTest,
ExcludeHiddenDocsWithoutNormsCountsOneTokenPerRow) {
+ index_query::testing::FakeIndexSource source;
+ source.set_doc_count(5);
+ index_query::IndexScoringStats stats {
+ .doc_count = 5, .token_count = 9, .has_positions = false,
.has_norms = false};
+ const std::vector<uint32_t> hidden {0, 4};
+ roaring::Roaring nulls;
+ nulls.add(1);
+ ASSERT_TRUE(exclude_hidden_docs(source, hidden, nulls, 0, &stats).ok());
+ EXPECT_EQ(stats.doc_count, 3);
+ EXPECT_EQ(stats.token_count, 3);
+}
+
+TEST(CollectionStatisticsHiddenRowsTest,
ExcludeHiddenDocsRejectsRowBeyondIndex) {
+ index_query::testing::FakeIndexSource source;
+ source.set_doc_count(2);
+ source.norms = {{0, 1}, {1, 1}};
+ index_query::IndexScoringStats stats {
+ .doc_count = 2, .token_count = 2, .has_positions = true,
.has_norms = true};
+ const std::vector<uint32_t> hidden {2};
+ Status status = exclude_hidden_docs(source, hidden, roaring::Roaring {},
2, &stats);
+ EXPECT_EQ(status.code(), ErrorCode::INVERTED_INDEX_FILE_CORRUPTED) <<
status;
+ const std::vector<uint32_t> nulls {2};
+ uint64_t non_null_length_sum = 0;
+ status = sum_non_null_norm_lengths(source, nulls, &non_null_length_sum);
+ EXPECT_EQ(status.code(), ErrorCode::INVERTED_INDEX_FILE_CORRUPTED) <<
status;
+}
+
+// A NULL row is a document without tokens, though its norm stands for one: it
stays in the
+// document count and its length is left out, hidden or not.
+TEST(CollectionStatisticsHiddenRowsTest,
ExcludeHiddenDocsLeavesOutLengthsOfNullRows) {
+ index_query::testing::FakeIndexSource source;
+ source.set_doc_count(5);
+ source.norms = {{0, 3}, {1, 5}, {2, 1}, {3, 7}, {4, 1}};
+ const std::vector<uint32_t> null_doc_ids {2, 4};
+ uint64_t non_null_length_sum = 0;
+ ASSERT_TRUE(sum_non_null_norm_lengths(source, null_doc_ids,
&non_null_length_sum).ok());
+ EXPECT_EQ(non_null_length_sum, 15);
+ roaring::Roaring nulls;
+ nulls.addMany(null_doc_ids.size(), null_doc_ids.data());
+
+ index_query::IndexScoringStats stats {
+ .doc_count = 5, .token_count = 1000, .has_positions = true,
.has_norms = true};
+ ASSERT_TRUE(exclude_hidden_docs(source, {}, nulls, non_null_length_sum,
&stats).ok());
+ EXPECT_EQ(stats.doc_count, 5);
+ EXPECT_EQ(stats.token_count, 15);
+
+ stats = {.doc_count = 5, .token_count = 1000, .has_positions = true,
.has_norms = true};
+ const std::vector<uint32_t> hidden {1, 2};
+ ASSERT_TRUE(exclude_hidden_docs(source, hidden, nulls,
non_null_length_sum, &stats).ok());
+ EXPECT_EQ(stats.doc_count, 3);
+ EXPECT_EQ(stats.token_count, 10);
+}
+
+TEST(CollectionStatisticsHiddenRowsTest,
ExcludeHiddenTermDocsTakesHiddenHoldersOnly) {
+ index_query::testing::FakeIndexSource source;
+ source.set_doc_count(6);
+ source.add("alpha", std::vector<uint32_t> {0, 1, 3, 5});
+ const std::vector<uint32_t> hidden {1, 2, 5};
+ uint64_t doc_frequency = 4;
+ ASSERT_TRUE(exclude_hidden_term_docs(source, "alpha", hidden,
&doc_frequency).ok());
+ EXPECT_EQ(doc_frequency, 2);
+
+ uint64_t absent = 0;
+ ASSERT_TRUE(exclude_hidden_term_docs(source, "omega", hidden,
&absent).ok());
+ EXPECT_EQ(absent, 0);
+}
+
TEST_F(CollectionStatisticsTest,
LegacyV3SkipsMissingSegmentAfterCollectingAvailableStatistics) {
auto tablet_schema = create_legacy_v3_schema();
const std::string first_segment_path = test_dir_ + "/legacy_v3_0.dat";
@@ -1290,8 +1696,10 @@ TEST(TabletCollectionStatisticsTest,
CollectsOnceAndCopiesToEveryCaller) {
CollectionStatistics first;
CollectionStatistics second;
- EXPECT_TRUE(tablet_statistics.copy_to(nullptr, tablet_schema, {}, nullptr,
&first).ok());
- EXPECT_TRUE(tablet_statistics.copy_to(nullptr, tablet_schema, {}, nullptr,
&second).ok());
+ EXPECT_TRUE(
+ tablet_statistics.copy_to(nullptr, tablet_schema, {}, nullptr,
nullptr, &first).ok());
+ EXPECT_TRUE(
+ tablet_statistics.copy_to(nullptr, tablet_schema, {}, nullptr,
nullptr, &second).ok());
}
// SNII scoring requires only positions and norms; statistics come directly
from the stats block.
diff --git a/be/test/storage/segment/inverted_index_searcher_cache_test.cpp
b/be/test/storage/segment/inverted_index_searcher_cache_test.cpp
index ae1a86e04a8..9410a479c39 100644
--- a/be/test/storage/segment/inverted_index_searcher_cache_test.cpp
+++ b/be/test/storage/segment/inverted_index_searcher_cache_test.cpp
@@ -24,6 +24,7 @@
#include <memory>
#include <string>
#include <thread>
+#include <vector>
#include "common/status.h"
#include "gtest/gtest_pred_impl.h"
@@ -319,4 +320,77 @@ TEST_F(InvertedIndexSearcherCacheTest,
remove_element_only_in_table) {
delete index_searcher_cache;
}
+// The non-NULL norm length sum of a field is computed once per cached
searcher, kept apart from
+// the other fields' sums, and not kept when computing it fails.
+TEST(CluceneSearcherCacheValueTest,
NonNullNormLengthSumIsComputedOncePerField) {
+ CluceneSearcherCacheValue value(FulltextIndexSearcherPtr(nullptr), 0, 0);
+ int computed = 0;
+ auto compute = [&computed](uint64_t sum) {
+ return [&computed, sum](uint64_t* out) {
+ ++computed;
+ *out = sum;
+ return Status::OK();
+ };
+ };
+ uint64_t sum = 0;
+ ASSERT_TRUE(value.non_null_norm_length_sum(L"a", compute(7), &sum).ok());
+ EXPECT_EQ(sum, 7);
+ ASSERT_TRUE(value.non_null_norm_length_sum(L"a", compute(9), &sum).ok());
+ EXPECT_EQ(sum, 7);
+ ASSERT_TRUE(value.non_null_norm_length_sum(L"b", compute(9), &sum).ok());
+ EXPECT_EQ(sum, 9);
+ EXPECT_EQ(computed, 2);
+
+ auto fail = [&computed](uint64_t*) {
+ ++computed;
+ return Status::Corruption("bad norms");
+ };
+ EXPECT_FALSE(value.non_null_norm_length_sum(L"c", fail, &sum).ok());
+ ASSERT_TRUE(value.non_null_norm_length_sum(L"c", compute(3), &sum).ok());
+ EXPECT_EQ(sum, 3);
+ EXPECT_EQ(computed, 4);
+}
+
+// The norm length sum of a set of hidden rows is computed once per field and
set, and a few sets
+// per field are kept, the oldest given up first.
+TEST(CluceneSearcherCacheValueTest,
HiddenNormLengthSumIsComputedOncePerRowSet) {
+ CluceneSearcherCacheValue value(FulltextIndexSearcherPtr(nullptr), 0, 0);
+ int computed = 0;
+ auto compute = [&computed](uint64_t sum) {
+ return [&computed, sum](uint64_t* out) {
+ ++computed;
+ *out = sum;
+ return Status::OK();
+ };
+ };
+ const std::vector<uint32_t> rows {1, 5, 9};
+ const std::vector<uint32_t> other_rows {1, 5, 10};
+ uint64_t sum = 0;
+ ASSERT_TRUE(value.hidden_norm_length_sum(L"a", rows, compute(7),
&sum).ok());
+ EXPECT_EQ(sum, 7);
+ ASSERT_TRUE(value.hidden_norm_length_sum(L"a", rows, compute(8),
&sum).ok());
+ EXPECT_EQ(sum, 7);
+ ASSERT_TRUE(value.hidden_norm_length_sum(L"a", other_rows, compute(8),
&sum).ok());
+ EXPECT_EQ(sum, 8);
+ ASSERT_TRUE(value.hidden_norm_length_sum(L"b", rows, compute(9),
&sum).ok());
+ EXPECT_EQ(sum, 9);
+ EXPECT_EQ(computed, 3);
+
+ auto fail = [](uint64_t*) { return Status::Corruption("bad norms"); };
+ const std::vector<uint32_t> failing_rows {2};
+ EXPECT_FALSE(value.hidden_norm_length_sum(L"a", failing_rows, fail,
&sum).ok());
+ ASSERT_TRUE(value.hidden_norm_length_sum(L"a", failing_rows, compute(4),
&sum).ok());
+ EXPECT_EQ(sum, 4);
+ EXPECT_EQ(computed, 4);
+
+ // Field "a" holds {1,5,9}, {1,5,10} and {2}; two more sets give up
{1,5,9}.
+ const std::vector<uint32_t> fourth {3};
+ const std::vector<uint32_t> fifth {4};
+ ASSERT_TRUE(value.hidden_norm_length_sum(L"a", fourth, compute(1),
&sum).ok());
+ ASSERT_TRUE(value.hidden_norm_length_sum(L"a", fifth, compute(1),
&sum).ok());
+ ASSERT_TRUE(value.hidden_norm_length_sum(L"a", rows, compute(6),
&sum).ok());
+ EXPECT_EQ(sum, 6);
+ EXPECT_EQ(computed, 7);
+}
+
} // namespace doris::segment_v2
diff --git
a/regression-test/data/fault_injection_p0/test_bm25_score_statistics_searcher_io_context.out
b/regression-test/data/fault_injection_p0/test_bm25_score_statistics_searcher_io_context.out
new file mode 100644
index 00000000000..bcd881009df
--- /dev/null
+++
b/regression-test/data/fault_injection_p0/test_bm25_score_statistics_searcher_io_context.out
@@ -0,0 +1,103 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !scores --
+0 0.6931471
+1 0.6931471
+10 0.6931471
+101 0.6931471
+105 0.6931471
+109 0.6931471
+113 0.6931471
+117 0.6931471
+12 0.6931471
+121 0.6931471
+125 0.6931471
+129 0.6931471
+13 0.6931471
+133 0.6931471
+137 0.6931471
+14 0.6931471
+141 0.6931471
+145 0.6931471
+149 0.6931471
+153 0.6931471
+157 0.6931471
+16 0.6931471
+161 0.6931471
+165 0.6931471
+169 0.6931471
+17 0.6931471
+173 0.6931471
+177 0.6931471
+18 0.6931471
+181 0.6931471
+185 0.6931471
+189 0.6931471
+193 0.6931471
+197 0.6931471
+2 0.6931471
+20 0.6931471
+21 0.6931471
+22 0.6931471
+24 0.6931471
+25 0.6931471
+26 0.6931471
+28 0.6931471
+29 0.6931471
+30 0.6931471
+32 0.6931471
+33 0.6931471
+34 0.6931471
+36 0.6931471
+37 0.6931471
+38 0.6931471
+4 0.6931471
+40 0.6931471
+41 0.6931471
+42 0.6931471
+44 0.6931471
+45 0.6931471
+46 0.6931471
+48 0.6931471
+49 0.6931471
+5 0.6931471
+50 0.6931471
+52 0.6931471
+53 0.6931471
+54 0.6931471
+56 0.6931471
+57 0.6931471
+58 0.6931471
+6 0.6931471
+60 0.6931471
+61 0.6931471
+62 0.6931471
+64 0.6931471
+65 0.6931471
+66 0.6931471
+68 0.6931471
+69 0.6931471
+70 0.6931471
+72 0.6931471
+73 0.6931471
+74 0.6931471
+76 0.6931471
+77 0.6931471
+78 0.6931471
+8 0.6931471
+80 0.6931471
+81 0.6931471
+82 0.6931471
+84 0.6931471
+85 0.6931471
+86 0.6931471
+88 0.6931471
+89 0.6931471
+9 0.6931471
+90 0.6931471
+92 0.6931471
+93 0.6931471
+94 0.6931471
+96 0.6931471
+97 0.6931471
+98 0.6931471
+
diff --git a/regression-test/data/inverted_index_p0/test_bm25_score.out
b/regression-test/data/inverted_index_p0/test_bm25_score.out
index 5b8ede10f56..4dfdb19d126 100644
--- a/regression-test/data/inverted_index_p0/test_bm25_score.out
+++ b/regression-test/data/inverted_index_p0/test_bm25_score.out
@@ -42,7 +42,7 @@
3 3 this is a test
-- !sql --
-4 4 hello world 1.0925692
+4 4 hello world 0.8025915
-- !sql --
4 4 hello world
diff --git
a/regression-test/data/inverted_index_p0/test_bm25_score_replica_statistics.out
b/regression-test/data/inverted_index_p0/test_bm25_score_replica_statistics.out
new file mode 100644
index 00000000000..90163f13821
--- /dev/null
+++
b/regression-test/data/inverted_index_p0/test_bm25_score_replica_statistics.out
@@ -0,0 +1,6 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !emptied --
+
+-- !refilled --
+4 0.2876821
+
diff --git
a/regression-test/data/inverted_index_p0/test_score_filter_matrix.out
b/regression-test/data/inverted_index_p0/test_score_filter_matrix.out
index 5ffe7ab9e17..e580e4b4ae2 100644
--- a/regression-test/data/inverted_index_p0/test_score_filter_matrix.out
+++ b/regression-test/data/inverted_index_p0/test_score_filter_matrix.out
@@ -148,32 +148,32 @@
6 0.7187315
-- !V2_match_mow --
-2 0.64072424
-3 0.90232176
+2 0.60996956
+3 0.8713851
-- !V2_match_mow_topn_control --
-2 0.64072424
-3 0.90232176
+2 0.60996956
+3 0.8713851
-- !V2_search_mow --
-2 0.64072424
-3 0.90232176
+2 0.60996956
+3 0.8713851
-- !V2_search_mow_topn_control --
-2 0.64072424
-3 0.90232176
+2 0.60996956
+3 0.8713851
-- !V2_match_mow_after_delete --
-3 1.0217376
+3 1.1823694
-- !V2_match_mow_after_delete_topn_control --
-3 1.0217376
+3 1.1823694
-- !V2_search_mow_after_delete --
-3 1.0217376
+3 1.1823694
-- !V2_search_mow_after_delete_topn_control --
-3 1.0217376
+3 1.1823694
-- !V3_match_range --
1 0.4700036
@@ -324,32 +324,32 @@
6 0.7187315
-- !V3_match_mow --
-2 0.64072424
-3 0.90232176
+2 0.60996956
+3 0.8713851
-- !V3_match_mow_topn_control --
-2 0.64072424
-3 0.90232176
+2 0.60996956
+3 0.8713851
-- !V3_search_mow --
-2 0.64072424
-3 0.90232176
+2 0.60996956
+3 0.8713851
-- !V3_search_mow_topn_control --
-2 0.64072424
-3 0.90232176
+2 0.60996956
+3 0.8713851
-- !V3_match_mow_after_delete --
-3 1.0217376
+3 1.1823694
-- !V3_match_mow_after_delete_topn_control --
-3 1.0217376
+3 1.1823694
-- !V3_search_mow_after_delete --
-3 1.0217376
+3 1.1823694
-- !V3_search_mow_after_delete_topn_control --
-3 1.0217376
+3 1.1823694
-- !SNII_match_range --
1 0.4700036
@@ -502,32 +502,32 @@
6 0.7187315
-- !SNII_match_mow --
-2 0.64072424
-3 0.90232176
+2 0.60996956
+3 0.8713851
-- !SNII_match_mow_topn_control --
-2 0.64072424
-3 0.90232176
+2 0.60996956
+3 0.8713851
-- !SNII_search_mow --
-2 0.64072424
-3 0.90232176
+2 0.60996956
+3 0.8713851
-- !SNII_search_mow_topn_control --
-2 0.64072424
-3 0.90232176
+2 0.60996956
+3 0.8713851
-- !SNII_match_mow_after_delete --
-3 1.0217376
+3 1.1823694
-- !SNII_match_mow_after_delete_topn_control --
-3 1.0217376
+3 1.1823694
-- !SNII_search_mow_after_delete --
-3 1.0217376
+3 1.1823694
-- !SNII_search_mow_after_delete_topn_control --
-3 1.0217376
+3 1.1823694
-- !ann_control --
1 0.0
diff --git
a/regression-test/suites/fault_injection_p0/test_bm25_score_statistics_searcher_io_context.groovy
b/regression-test/suites/fault_injection_p0/test_bm25_score_statistics_searcher_io_context.groovy
new file mode 100644
index 00000000000..4e7f3f71ed9
--- /dev/null
+++
b/regression-test/suites/fault_injection_p0/test_bm25_score_statistics_searcher_io_context.groovy
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// BM25 statistics open the CLucene searcher of a segment the searcher cache
does not hold yet.
+// Once its meta is read, the searcher's stream must drop the opening query's
I/O context and stop
+// reading as index meta, as the query path's searcher does:
CSIndexInput.readInternal fails any
+// read of index data that still reads as meta. Loads must not put the
searchers into the cache
+// first, or the statistics would not open them.
+suite("test_bm25_score_statistics_searcher_io_context", "nonConcurrent") {
+ sql "DROP TABLE IF EXISTS test_bm25_score_statistics_searcher_io_context"
+ sql """
+ CREATE TABLE test_bm25_score_statistics_searcher_io_context (
+ id BIGINT,
+ content TEXT NULL,
+ INDEX idx_content (content) USING INVERTED PROPERTIES("parser" =
"english")
+ ) ENGINE=OLAP
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1",
+ "enable_unique_key_merge_on_write" = "true",
+ "disable_auto_compaction" = "true"
+ );
+ """
+ setBeConfigTemporary([enable_write_index_searcher_cache: false]) {
+ sql """
+ INSERT INTO test_bm25_score_statistics_searcher_io_context
+ SELECT number, concat_ws(' ', if(number % 4 = 1, 'jobs', 'work'),
'alpha') FROM numbers('number' = '200')
+ """
+ // Overwritten rows make the statistics read postings of the hidden
rows.
+ sql """
+ INSERT INTO test_bm25_score_statistics_searcher_io_context
+ SELECT number * 2, 'jobs again' FROM numbers('number' = '50')
+ """
+ sql "sync"
+ }
+
+ try {
+ GetDebugPoint().enableDebugPointForAllBEs("CSIndexInput.readInternal")
+ order_qt_scores """
+ SELECT id, score() FROM
test_bm25_score_statistics_searcher_io_context
+ WHERE content MATCH_ANY 'jobs' ORDER BY score() DESC LIMIT 1000
+ """
+ } finally {
+ GetDebugPoint().disableDebugPointForAllBEs("CSIndexInput.readInternal")
+ }
+}
diff --git
a/regression-test/suites/inverted_index_p0/test_bm25_score_replica_statistics.groovy
b/regression-test/suites/inverted_index_p0/test_bm25_score_replica_statistics.groovy
new file mode 100644
index 00000000000..6494330d4ba
--- /dev/null
+++
b/regression-test/suites/inverted_index_p0/test_bm25_score_replica_statistics.groovy
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import java.util.concurrent.TimeUnit
+import org.awaitility.Awaitility
+
+// A merge-on-write replica keeps overwritten rows and rows written with the
delete sign in its
+// segments until it compacts them, and each replica compacts on its own
schedule. BM25 statistics
+// must count only the rows a query sees, so a row scores the same on every
replica, before and
+// after compaction, and the same as in a table freshly loaded with the
visible rows.
+suite("test_bm25_score_replica_statistics", "p0") {
+ def aliveBackends = sql_return_maparray("SHOW BACKENDS").findAll {
it.Alive.toString() == "true" }
+ def replicaNum = aliveBackends.size() >= 3 ? 3 : 1
+
+ def createTable = { String name, String keys, int replicas ->
+ sql "DROP TABLE IF EXISTS ${name}"
+ def mergeOnWrite = keys.startsWith("UNIQUE") ?
'"enable_unique_key_merge_on_write" = "true",' : ""
+ sql """
+ CREATE TABLE ${name} (
+ id BIGINT,
+ content TEXT NULL,
+ INDEX idx_content (content) USING INVERTED
+ PROPERTIES("parser" = "english", "support_phrase" = "true")
+ ) ENGINE=OLAP
+ ${keys}
+ DISTRIBUTED BY HASH(id) BUCKETS 4
+ PROPERTIES (
+ "replication_num" = "${replicas}",
+ ${mergeOnWrite}
+ "disable_auto_compaction" = "true"
+ );
+ """
+ }
+ createTable("test_bm25_score_replica_statistics", "UNIQUE KEY(id)",
replicaNum)
+ createTable("test_bm25_score_replica_statistics_fresh", "UNIQUE KEY(id)",
1)
+
+ // Documents of different lengths, some longer than the lengths norms hold
exactly, and NULLs.
+ def text = { int round ->
+ """if((number + ${round}) % 7 = 0, NULL, concat_ws(' ',
+ repeat('alpha beta gamma ', cast((number * 7 + ${round} * 13) %
15 AS INT)),
+ if((number + ${round}) % 3 = 0, 'jobs', 'work'),
+ if((number * 5 + ${round}) % 4 = 0, 'try hard', 'rest'),
+ repeat('delta ', cast((number + ${round}) % 5 AS INT))))"""
+ }
+ for (int batch = 0; batch < 6; batch++) {
+ sql "INSERT INTO test_bm25_score_replica_statistics SELECT number +
${batch} * 500, ${text(batch)} FROM numbers('number' = '500')"
+ }
+ // Overwrite a quarter of the rows three times.
+ for (int round = 1; round <= 3; round++) {
+ sql "INSERT INTO test_bm25_score_replica_statistics SELECT number * 4
+ ${round}, ${text(round + 20)} FROM numbers('number' = '750')"
+ }
+ // Delete rows with DELETE, and with full rows carrying the delete sign,
as CDC loads write them.
+ sql "DELETE FROM test_bm25_score_replica_statistics WHERE id % 17 = 3"
+ sql "SET show_hidden_columns = true"
+ sql """
+ INSERT INTO test_bm25_score_replica_statistics (id, content,
__DORIS_DELETE_SIGN__)
+ SELECT number * 13 + 5, ${text(40)}, 1 FROM numbers('number' = '200')
+ """
+ sql "SET show_hidden_columns = false"
+ sql "INSERT INTO test_bm25_score_replica_statistics_fresh SELECT id,
content FROM test_bm25_score_replica_statistics"
+ sql "sync"
+
+ def scores = { String table, String vars ->
+ def rows = sql """
+ SELECT /*+ SET_VAR(${vars}) */ id, score()
+ FROM ${table}
+ WHERE SEARCH('content:jobs OR content:try')
+ ORDER BY score() DESC
+ LIMIT 100000
+ """
+ def byId = [:]
+ rows.each { byId[it[0]] = it[1] }
+ return byId
+ }
+ def assertSameScores = { Map expected, Map actual, String state ->
+ assertTrue(expected.size() > 0)
+ assertEquals(expected.size(), actual.size())
+ def differing = expected.findAll { id, score -> actual[id] != score }
+ assertTrue(differing.isEmpty(),
+ "${state}: ${differing.size()} rows score differently, "
+ + "e.g. ${differing.take(3).collect { id, score -> "id
${id}: ${actual[id]} vs ${score}" }}")
+ }
+
+ def expected = scores("test_bm25_score_replica_statistics_fresh",
"use_fix_replica = -1")
+ def assertScoresLikeFreshTable = { String vars, String state ->
+ assertSameScores(expected,
scores("test_bm25_score_replica_statistics", vars),
+ "${state} (${vars}), compared with a freshly loaded table")
+ }
+
+ assertScoresLikeFreshTable("use_fix_replica = -1", "before compaction")
+
+ // Cloud mode keeps one shared copy of each tablet, so there are no
replicas to compact apart,
+ // and its compaction is driven differently.
+ if (!isCloudMode() && replicaNum == 3) {
+ // Compact the replicas on one backend only, so the replicas hold
different rows.
+ def backendIdToIp = [:]
+ def backendIdToHttpPort = [:]
+ getBackendIpHttpPort(backendIdToIp, backendIdToHttpPort)
+ def tablets = sql_return_maparray("SHOW TABLETS FROM
test_bm25_score_replica_statistics")
+ def compactedBackend = tablets[0].BackendId
+ def compactedTablets = tablets.findAll { it.BackendId ==
compactedBackend }
+ for (tablet in compactedTablets) {
+ def (code, out, err) =
be_run_full_compaction(backendIdToIp["${compactedBackend}"],
+ backendIdToHttpPort["${compactedBackend}"],
tablet.TabletId)
+ assertEquals(0, code)
+ assertTrue(out.toLowerCase().contains("success"), out)
+ }
+ Awaitility.await().atMost(300, TimeUnit.SECONDS).pollInterval(1,
TimeUnit.SECONDS).until {
+ compactedTablets.every { tablet ->
+ def (code, out, err) =
be_show_tablet_status(backendIdToIp["${compactedBackend}"],
+ backendIdToHttpPort["${compactedBackend}"],
tablet.TabletId)
+ code == 0 && parseJson(out.trim()).rowsets.size() == 1
+ }
+ }
+
+ for (int run = 0; run < 10; run++) {
+ assertScoresLikeFreshTable("use_fix_replica = -1", "one replica
compacted, run ${run}")
+ }
+ for (int replica = 0; replica < 3; replica++) {
+ assertScoresLikeFreshTable("use_fix_replica = ${replica}", "one
replica compacted")
+ }
+ }
+
+ if (!isCloudMode()) {
+ trigger_and_wait_compaction("test_bm25_score_replica_statistics",
"full")
+ for (int replica = 0; replica < replicaNum; replica++) {
+ assertScoresLikeFreshTable("use_fix_replica = ${replica}", "all
replicas compacted")
+ }
+ }
+
+ // NULL rows have no tokens, on a merge-on-write table as on a duplicate
table: with lengths the
+ // norms hold exactly, both score alike.
+ createTable("test_bm25_score_replica_statistics_nulls_mow", "UNIQUE
KEY(id)", 1)
+ createTable("test_bm25_score_replica_statistics_nulls_dup", "DUPLICATE
KEY(id)", 1)
+ for (table in ["test_bm25_score_replica_statistics_nulls_mow",
"test_bm25_score_replica_statistics_nulls_dup"]) {
+ sql """
+ INSERT INTO ${table}
+ SELECT number, if(number % 2 = 0, NULL,
+ concat_ws(' ', if(number % 4 = 1, 'jobs',
'work'), repeat('alpha ', cast(number % 6 AS INT))))
+ FROM numbers('number' = '1000')
+ """
+ }
+ sql "sync"
+ assertSameScores(scores("test_bm25_score_replica_statistics_nulls_dup",
"use_fix_replica = -1"),
+ scores("test_bm25_score_replica_statistics_nulls_mow",
"use_fix_replica = -1"),
+ "merge-on-write table with NULL rows, compared with a duplicate
table")
+
+ // A bucket whose rows are all deleted still has segments but no row a
query can match. Scoring
+ // must return nothing for it instead of failing.
+ createTable("test_bm25_score_replica_statistics_emptied", "UNIQUE
KEY(id)", 1)
+ sql "INSERT INTO test_bm25_score_replica_statistics_emptied VALUES (1, 'we
should try these jobs'), (2, 'more jobs'), (3, 'other')"
+ sql "DELETE FROM test_bm25_score_replica_statistics_emptied WHERE id > 0"
+ order_qt_emptied """
+ SELECT id, score() FROM test_bm25_score_replica_statistics_emptied
+ WHERE SEARCH('content:jobs') ORDER BY score() DESC LIMIT 10
+ """
+ sql "INSERT INTO test_bm25_score_replica_statistics_emptied VALUES (4,
'jobs again')"
+ order_qt_refilled """
+ SELECT id, score() FROM test_bm25_score_replica_statistics_emptied
+ WHERE SEARCH('content:jobs') ORDER BY score() DESC LIMIT 10
+ """
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
