Harnoor7 commented on code in PR #14479:
URL: https://github.com/apache/pinot/pull/14479#discussion_r1848755543
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java:
##########
@@ -797,6 +801,18 @@ private void addNewRow(int docId, GenericRow row) {
recordIndexingError(indexEntry.getKey(), e);
}
}
+
+ if (_thresholdForNumOfColValuesEnabled) {
+ int prevCount = indexContainer._valuesInfo.getNumValues();
+ long newCount = prevCount + 1L + values.length;
Review Comment:
Mutable index is unbounded right (No code enforced limit)? From the code I
see Realtime Mutable index is always created with dictionary (even for MV
VarByte col with noDict enabled in config).
Hence from `numOfValues` we know that size of mutable index is approx:
`numOfValues * 4 Bytes`.
But we are more interested in the size of immutable index since that's where
exception is being thrown. However Immutable index can be larger or even
smaller than mutable index as implementation is completely diff. So while
building mutable index we need to keep some state to estimate the approx size
of the immutable version of the index (Like while building mutable fwd index,
we need to keep an estimation of bitmap, numBitsPerValue, header size, etc).
> We should perform a per-index check (add an api to the MutableIndex and
let it return if it can take more values).
So this might be adding too much complexity to every mutable index since now
after every row consumption we need to update estimated size of the
corresponding immutable index and it tight couples code with immutable index
logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
