Jackie-Jiang commented on a change in pull request #6409:
URL: https://github.com/apache/incubator-pinot/pull/6409#discussion_r557018411
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/BitmapInvertedIndexReader.java
##########
@@ -18,84 +18,71 @@
*/
package org.apache.pinot.core.segment.index.readers;
-import com.google.common.base.Preconditions;
import java.lang.ref.SoftReference;
+import java.nio.ByteOrder;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+/**
+ * Reader for bitmap based inverted index. Please reference
+ * {@link
org.apache.pinot.core.segment.creator.impl.inv.BitmapInvertedIndexWriter} for
the index file layout.
+ */
public class BitmapInvertedIndexReader implements
InvertedIndexReader<ImmutableRoaringBitmap> {
public static final Logger LOGGER =
LoggerFactory.getLogger(BitmapInvertedIndexReader.class);
- private final PinotDataBuffer _dataBuffer;
private final int _numBitmaps;
+ private final PinotDataBuffer _offsetBuffer;
+ private final PinotDataBuffer _bitmapBuffer;
+
+ // Use the offset of the first bitmap to support 2 different format of the
inverted index:
+ // 1. Offset buffer stores the offsets within the whole data buffer
(including offset buffer)
+ // 2. Offset buffer stores the offsets within the bitmap buffer
+ private final int _firstOffset;
private volatile SoftReference<SoftReference<ImmutableRoaringBitmap>[]>
_bitmaps;
- /**
- * Constructs an inverted index with the specified size.
- * @param dataBuffer data buffer for the inverted index.
- * @param cardinality the number of bitmaps in the inverted index, which
should be the same as the
- * number of values in
- * the dictionary.
- */
- public BitmapInvertedIndexReader(PinotDataBuffer dataBuffer, int
cardinality) {
- _dataBuffer = dataBuffer;
- _numBitmaps = cardinality;
+ public BitmapInvertedIndexReader(PinotDataBuffer dataBuffer, int numBitmaps)
{
+ _numBitmaps = numBitmaps;
- int lastOffset = _dataBuffer.getInt(_numBitmaps * Integer.BYTES);
- Preconditions.checkState(lastOffset == _dataBuffer.size(),
- "The last offset should be equal to buffer size! Current lastOffset: "
+ lastOffset + ", buffer size: "
- + _dataBuffer.size());
+ long offsetBufferEndOffset = (long) (numBitmaps + 1) * Integer.BYTES;
+ _offsetBuffer = dataBuffer.view(0, offsetBufferEndOffset,
ByteOrder.BIG_ENDIAN);
+ _bitmapBuffer = dataBuffer.view(offsetBufferEndOffset, dataBuffer.size());
Review comment:
Yes, to be able to read the bitmap index within the H3 index
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
