jackjlli commented on pull request #6107: URL: https://github.com/apache/incubator-pinot/pull/6107#issuecomment-703921970
The config `segmentPushType` is needed at least in the following scenarios: 1. Retention manager needs it to purge segments: https://github.com/apache/incubator-pinot/blob/9929dad2e804f525b8ace925e859e07c6fdee9da/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java#L104 2. `TableRetentionValidator` gets called https://github.com/apache/incubator-pinot/blob/fb043dab7d5fef3ef726896d2f53e45a2fd69f22/pinot-controller/src/main/java/org/apache/pinot/controller/util/TableRetentionValidator.java#L112 3. Segment creation Hadoop/Spark job requires it to validate the time column: https://github.com/apache/incubator-pinot/blob/fb043dab7d5fef3ef726896d2f53e45a2fd69f22/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java#L145 4. Segment creation job requires it to generate the segment name: https://github.com/apache/incubator-pinot/blob/540853d786c9e4b4bb27bf4bbc6fa73837ff1bc3/pinot-core/src/main/java/org/apache/pinot/core/segment/name/NormalizedDateSegmentNameGenerator.java#L48 So if you remove the check, you'd also need to adjust those logic to avoid breaking the current behaviors. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
