FrankMormino commented on issue #66365:
URL: https://github.com/apache/doris/issues/66365#issuecomment-6152084604
We hit what looks like the same bug on **4.1.3** (release build) and have a
CPU profile that points to a root cause: an infinite loop in the RLE decoder
for BOOLEAN columns during the lazy-materialization row fetch. The relevant
code is unchanged in 4.1.4.1 and master.
### Symptom
Scan threads of the workload group's remote-scan pool (`rs_normal`, 16
threads) get stuck one at a time at 100% CPU and never return. The query is
cancelled at `query_timeout` (900 s), but the thread keeps spinning. Over about
12 days all 16 threads were stuck and the BE sat at ~1560% CPU. Only a BE
restart frees them. The tables have many BOOLEAN / `tinyint(1)` columns (Unique
Key MoW, Kafka-sink loaded); the trigger is ad hoc `SELECT ... ORDER BY ...
LIMIT n` (n <= 1024).
### Stack (gperftools `/pprof/profile`, 10 s; every hot sample is in these
frames)
```
BitReader::GetVlqInt
RleDecoder<bool>::ReadHeader
RleDecoder<bool>::Skip
RlePageDecoder<FieldType::OLAP_FIELD_TYPE_BOOL>::read_by_rowids
FileColumnIterator::read_by_rowids
RowIdStorageReader::read_by_rowids
PInternalService::multiget_data_v2
```
### Why it never ends
`be/src/util/rle_encoding.h`, `RleDecoder<T>::Skip`:
```cpp
while (to_skip > 0) {
bool result = ReadHeader();
DCHECK(result);
...
}
```
Once the bit reader is exhausted, `ReadHeader()` returns false and leaves
`repeat_count_` / `literal_count_` at 0, so `to_skip` never decreases. `DCHECK`
is a no-op in release builds, and there is no cancellation check, so the loop
spins forever.
Likely entry: `be/src/storage/segment/rle_page.h`,
`RlePageDecoder::read_by_rowids`:
```cpp
ordinal_t ord = rowids[i] - page_first_ordinal;
...
_rle_decoder.Skip(ord - _cur_index);
```
If row ids reach the page out of order or duplicated (`ord < _cur_index`),
the unsigned `ord - _cur_index` underflows to a huge value and `Skip` runs past
the end of the page. In `rowid_fetcher.cpp` the internal-table path groups row
ids by consecutive file id but does not sort or dedup them within a segment;
the external-table path does both.
### Workaround (confirmed)
`SET GLOBAL topn_lazy_materialization_threshold = -1;` (or `0`, as in this
issue), then restart the BE to free the threads that are already stuck. 4.1.4.1
alone does not help.
### Suggested fix
1. `RleDecoder::Skip` / `Get`: stop (return 0 / false) when `ReadHeader()`
fails instead of looping.
2. `RlePageDecoder::read_by_rowids`: handle `ord < _cur_index` (re-seek the
decoder) instead of skipping by an underflowed delta.
3. Sort and dedup row ids per segment in the internal multiget path, as the
external path does.
4. Add a cancellation check to the multiget read loop so a stuck fetch
cannot outlive its query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
