rohityadav1993 commented on code in PR #13837:
URL: https://github.com/apache/pinot/pull/13837#discussion_r1744308078
##########
pinot-connectors/pinot-flink-connector/src/main/java/org/apache/pinot/connector/flink/sink/FlinkSegmentWriter.java:
##########
@@ -124,12 +135,26 @@ public void init(TableConfig tableConfig, Schema schema,
Map<String, String> bat
"batchConfigMaps must contain only 1 BatchConfig for table: %s",
_tableNameWithType);
Map<String, String> batchConfigMap =
_batchIngestionConfig.getBatchConfigMaps().get(0);
+ batchConfigMap.put(BatchConfigProperties.UPLOADED_REALTIME_PARTITION_ID,
Integer.toString(_indexOfSubtask));
+ batchConfigMap.put(BatchConfigProperties.SEGMENT_UPLOAD_TIME_MS,
String.valueOf(_segmentUploadTimeMs));
+ batchConfigMap.computeIfAbsent(
+ BatchConfigProperties.SEGMENT_NAME_GENERATOR_PROP_PREFIX + "." +
BatchConfigProperties.SEGMENT_NAME_PREFIX,
Review Comment:
Makes sense, mandating UploadedRealtimeSegmentName naming can be only kept
for upsert table which didn't work before. Will update.
Though, I do think we can make UploadedRealtimeSegmentName as default naming
convention for uploaded segments for non upsert as well as it will lead to a
better segment assignment which currently happens by taking mod with 10_000
```
if (segmentPartitionId == null) {
// This case is for the uploaded segments for which there's no
partition information.
// A random, but consistent, partition id is calculated based on the
hash code of the segment name.
// Note that '% 10K' is used to prevent having partition ids with
large value which will be problematic later in
// instance assignment formula.
segmentPartitionId = Math.abs(segmentName.hashCode() % 10_000);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
