KKcorps commented on code in PR #9802:
URL: https://github.com/apache/pinot/pull/9802#discussion_r1032761596
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java:
##########
@@ -156,7 +178,43 @@ private DimensionTable createDimensionTable() {
}
}
}
- return new DimensionTable(schema, primaryKeyColumns, lookupTable);
+ return new FastLookupDimensionTable(schema, primaryKeyColumns,
lookupTable);
+ } finally {
+ for (SegmentDataManager segmentManager : segmentManagers) {
+ releaseSegment(segmentManager);
+ }
+ }
+ }
+
+ private DimensionTable createMemOptimisedDimensionTable() {
+ Schema schema = ZKMetadataProvider.getTableSchema(_propertyStore,
_tableNameWithType);
+ Preconditions.checkState(schema != null, "Failed to find schema for
dimension table: %s", _tableNameWithType);
+
+ List<String> primaryKeyColumns = schema.getPrimaryKeyColumns();
+ Preconditions.checkState(CollectionUtils.isNotEmpty(primaryKeyColumns),
+ "Primary key columns must be configured for dimension table: %s",
_tableNameWithType);
+
+ Map<PrimaryKey, LookupRecordLocation> lookupTable = new HashMap<>();
+ List<SegmentDataManager> segmentManagers = acquireAllSegments();
+ try {
+ for (SegmentDataManager segmentManager : segmentManagers) {
+ IndexSegment indexSegment = segmentManager.getSegment();
+ int numTotalDocs = indexSegment.getSegmentMetadata().getTotalDocs();
+ if (numTotalDocs > 0) {
+ try (PinotSegmentRecordReader recordReader = new
PinotSegmentRecordReader()) {
Review Comment:
My bad. I have changed the logic now to close the segments only when
shutting down the table data manager. The record reader is closed when we
replace the table we newer instance. Both of these happen only when preload is
disabled, otherwise old logic is applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
