Jackie-Jiang commented on code in PR #18496:
URL: https://github.com/apache/pinot/pull/18496#discussion_r3243171725
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -255,33 +257,64 @@ private Map<String, String>
createForwardIndexForSVColumn()
(BitmapInvertedIndexReader) InvertedIndexType.ReaderFactory
.INSTANCE.createSkippingForward(_segmentWriter, _columnMetadata);
Dictionary dictionary = DictionaryIndexType.read(_segmentWriter,
_columnMetadata)) {
- // Construct the forward index in the values buffer
+ // Construct the forward index in the values buffer. For var-length
columns, also gather per-element stats
+ // (lengthOfShortest/Longest, isAscii for STRING) inline when the source
segment is missing them, so the
+ // backfill happens without a second dictionary scan.
+ DataType storedType = _columnMetadata.getStoredType();
+ boolean backfillStats =
+ !storedType.isFixedWidth() &&
_columnMetadata.getLengthOfShortestElement() < 0;
Review Comment:
Added `testBackfillFromInvertedIndexRebuild` in `ForwardIndexHandlerTest`.
It strips the 1.6.0-era stats from `DIM_SV_FORWARD_INDEX_DISABLED_STRING` (SV)
and `DIM_MV_FORWARD_INDEX_DISABLED_STRING` (MV), drops `FORWARD_INDEX_DISABLED`
from the field config so `updateIndices` queues `ENABLE_DICT_FORWARD_INDEX`,
and verifies that `InvertedIndexAndDictionaryBasedForwardIndexCreator`'s inline
backfill persists all four stats — including `MAX_ROW_LENGTH_IN_BYTES` for the
MV case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
