github-actions[bot] commented on code in PR #69438:
URL: https://github.com/apache/doris/pull/69438#discussion_r4365847990
##########
be/src/storage/index/inverted/reader/collection_statistics.cpp:
##########
@@ -361,6 +556,33 @@ Status CollectionStatistics::process_segment(const
RowsetSharedPtr& rowset,
ws_field_name, io_ctx);
index_query::IndexScoringStats scoring_stats;
RETURN_IF_ERROR(source->scoring_stats(&scoring_stats));
+ if (hidden_rows != nullptr) {
+ std::shared_ptr<roaring::Roaring> null_docs =
std::make_shared<roaring::Roaring>();
+ uint64_t non_null_length_sum = 0;
+ if (scoring_stats.has_norms) {
+ segment_v2::InvertedIndexQueryCacheHandle
null_bitmap_cache_handle;
+ RETURN_IF_ERROR(segment_v2::read_clucene_null_bitmap(
+ *idx_file_reader, *collect_info.index_meta, io_ctx,
+ &null_bitmap_cache_handle));
+ null_docs = null_bitmap_cache_handle.get_bitmap();
+ DORIS_CHECK(null_docs != nullptr);
+ auto sum_lengths = [&source, &null_docs](uint64_t* out) {
+ std::vector<uint32_t>
null_doc_ids(null_docs->cardinality());
+ null_docs->toUint32Array(null_doc_ids.data());
+ return
collection_statistics_detail::sum_non_null_norm_lengths(
+ *source, null_doc_ids, out);
+ };
+#ifdef BE_TEST
+ RETURN_IF_ERROR(sum_lengths(&non_null_length_sum));
+#else
+
RETURN_IF_ERROR(inverted_index_cache_handle.get<CluceneSearcherCacheValue>()
Review Comment:
[P1] Account for the norms retained by statistics collection
This callback now loads norms for every segment in a MoW tablet's captured
read set before normal key/zone-map pruning, even when its hidden-row set is
empty. CLucene's `SegmentReader::norms()` allocates and retains one byte per
document per scored field, but these searchers were inserted using only
`getTermInfosRAMUsed()` and neither their cache charge nor tracked bytes are
updated afterward. A selective query over a large, low-vocabulary tablet can
therefore leave gigabytes of norms resident while the searcher cache accounts
for only small dictionaries; repeated queries across tablets can exhaust BE
memory despite the configured cache limit. Previously such pruned segments only
contributed dictionary statistics and did not load these norms. Please
reserve/track the added buffers and update the cache charge, including for
searchers already in the cache, or compute the sum without retaining
unaccounted norm buffers.
##########
be/src/storage/index/inverted/query/exec/index_source.cpp:
##########
@@ -43,4 +46,128 @@ Status IndexSource::collect_terms(std::span<const
std::string> terms, DocIdSink&
return Status::OK();
}
+namespace {
+
+// How many postings a scan decodes in the time a seek costs.
+constexpr uint64_t kPostingsPerSeek = 32;
+
+// Decodes every posting once and tests it against a bitset of `docs`.
+Status count_listed_docs_by_scan(PostingsCursor& cursor, std::span<const
uint32_t> docs,
+ uint64_t* out) {
+ std::vector<uint64_t> bits(docs.back() / 64 + 1, 0);
Review Comment:
[P2] Avoid a segment-sized bitmap for sparse term intersections
The scan branch is selected from `docs.size()` and `doc_freq()`, but this
allocation scales with the largest hidden row ID. For example, `hidden_docs =
{9999999}` and a term with one posting select this branch and zero 1,250,000
bytes just to compare two IDs. CLucene also returns a non-null cursor for an
absent term, so `df = 0` incurs the same allocation before discovering EOF.
Collection statistics repeat this for every query term and every segment with
hidden rows on every query, making selective or many-term searches perform
substantial segment-sized allocation/zeroing even with tiny or empty posting
lists. Please use a sparse/sorted intersection for this case, account for the
maximum ID when choosing a dense bitset, and short-circuit zero-frequency terms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
