Jackie-Jiang commented on code in PR #8721:
URL: https://github.com/apache/pinot/pull/8721#discussion_r875303678
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java:
##########
@@ -128,25 +128,26 @@ private void loadLookupTable()
} while (!UPDATER.compareAndSet(this, snapshot, replacement));
}
- private DimensionTable createDimensionTable()
- throws Exception {
- Map<PrimaryKey, GenericRow> map = new HashMap<>();
+ private DimensionTable createDimensionTable() {
+ Schema schema = ZKMetadataProvider.getTableSchema(_propertyStore,
_tableNameWithType);
+ Preconditions.checkState(schema != null, "Failed to find schema for
dimension table: %s", _tableNameWithType);
+ List<String> primaryKeyColumns = schema.getPrimaryKeyColumns();
+ Preconditions.checkState(CollectionUtils.isNotEmpty(primaryKeyColumns),
+ "Primary key columns must be configured for dimension table: %s",
_tableNameWithType);
+
+ Map<PrimaryKey, GenericRow> lookupTable = new HashMap<>();
List<SegmentDataManager> segmentManagers = acquireAllSegments();
try {
for (SegmentDataManager segmentManager : segmentManagers) {
IndexSegment indexSegment = segmentManager.getSegment();
- try (PinotSegmentRecordReader reader = new PinotSegmentRecordReader(
- indexSegment.getSegmentMetadata().getIndexDir())) {
- while (reader.hasNext()) {
- GenericRow row = reader.next();
- map.put(row.getPrimaryKey(_dimensionTable.getPrimaryKeyColumns()),
row);
- }
+ int numTotalDocs = indexSegment.getSegmentMetadata().getTotalDocs();
Review Comment:
Good question. We don't want to create a separate record reader because that
will load the segment again. Instead, we can directly read the record out of
the already loaded segment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
