rohityadav1993 commented on code in PR #14443:
URL: https://github.com/apache/pinot/pull/14443#discussion_r1841713786
##########
pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common/src/main/java/org/apache/pinot/plugin/ingestion/batch/common/SegmentGenerationTaskRunner.java:
##########
@@ -62,6 +62,10 @@ public class SegmentGenerationTaskRunner implements
Serializable {
public static final String FILE_PATH_PATTERN = "file.path.pattern";
public static final String SEGMENT_NAME_TEMPLATE = "segment.name.template";
+ // For UploadedRealtimeSegmentNameGenerator
+ public static final String SEGMENT_PARTITION_ID = "segment.partitionId";
+ public static final String SEGMENT_UPLOAD_TIME_MS = "segment.uploadTimeMs";
Review Comment:
`SegmentGeneratorConfig.segmentCreationTime` can be reused as
`SEGMENT_UPLOAD_TIME`. Every run of ingestion job should have a time associated
with it and should populate it.
Since Flink can also be run in batch mode, e.g. hourly/daily; start/end time
can be used as `creationTime`. This provides deterministic segment names and
one run's segments will have same timestamp in segment name.
Flink uses table config but also overrides transient properties like
partitionId, seqId, creationTime in
[FlinkSegmentWriter](https://github.com/apache/pinot/pull/13837/files#diff-72b89e65ea292a808b5d8a902d8a64adb87834534414d5b9d0e46c4725a0455cL126)
through flink job config `SEGMENT_UPLOAD_TIME_MS`. This way fixed
properties(like prefix) can be part of table config and transient properties
should come from the ingestion job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
