MrNeocore opened a new issue #7090: URL: https://github.com/apache/incubator-pinot/issues/7090
Following Slack discussion: https://apache-pinot.slack.com/archives/C011C9JHN7R/p1624443889243500 **Use case:** Some tables in Pinot are used in conjunction with [IdSet filtering](https://docs.google.com/document/d/1s6DZ9eTPqH7vaKQlPjKiWb_OBC3hkkEGICIzcd5gozc/edit#) or [Lookups](https://docs.pinot.apache.org/users/user-guide-query/lookup-udf-join) - and in some cases don't have a time column. - Hence time-based segment name generation strategies do not allow for simple segment replacements whenever data in those "dimension" tables change. **Proposition** Allow segment name generation to be based on the input file names such that segments can be named following a user provided id. E.g. ``` basedir/id1/file.parquet basedir/id2/file.parquet ``` Would generate segments ``` <table_name>_id1.segment <table_name>_id2.segment ``` Currently, the [SegmentNameGenerator interface](https://github.com/apache/incubator-pinot/blob/7ce8b756dff2b153151517afd552759e44d46c75/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/name/SegmentNameGenerator.java) doesn't allow input file names, therefore it is not possible to implement a strategy similar to the one presented above. **Note**: If you known any other alternative to reach the use case goal, please feel free to provide ideas ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
