ankitsultana commented on code in PR #14742:
URL: https://github.com/apache/pinot/pull/14742#discussion_r1901356989
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompactmerge/UpsertCompactMergeTaskGenerator.java:
##########
@@ -296,18 +307,24 @@ public static SegmentSelectionResult
processValidDocIdsMetadata(Map<String, Stri
// variables to maintain current group sum
long currentValidDocsSum = 0;
long currentTotalDocsSum = 0;
+ double currentOutputSegmentSizeInBytes = 0.0;
for (SegmentMergerMetadata segment : segments) {
long validDocs = segment.getValidDocIds();
long invalidDocs = segment.getInvalidDocIds();
+ double expectedSegmentSizeInBytes =
+ (segment.getSegmentZKMetadata().getSizeThresholdToFlushSegment() *
validDocs * 1.0)
Review Comment:
`getSizeThresholdToFlushSegment` returns the number of rows not the size in
bytes.
##########
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/upsertcompactmerge/UpsertCompactMergeTaskGenerator.java:
##########
@@ -408,6 +426,10 @@ public void validateTaskConfigs(TableConfig tableConfig,
Map<String, String> tas
Preconditions.checkState(upsertConfig.isEnableSnapshot(),
String.format("'enableSnapshot' from UpsertConfig must be enabled for
%s",
MinionConstants.UpsertCompactMergeTask.TASK_TYPE));
+ // check valid task config for maxOutputSegmentSize
+ if
(taskConfigs.containsKey(MinionConstants.UpsertCompactMergeTask.MAX_NUM_RECORDS_PER_SEGMENT_KEY))
{
+
DataSizeUtils.toBytes(taskConfigs.get(MinionConstants.UpsertCompactMergeTask.MAX_NUM_RECORDS_PER_SEGMENT_KEY));
Review Comment:
bug? the key is max num records
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
