swaminathanmanish commented on code in PR #14425:
URL: https://github.com/apache/pinot/pull/14425#discussion_r1845494692
##########
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/mapper/SegmentMapper.java:
##########
@@ -71,21 +71,28 @@ public class SegmentMapper {
private final boolean _includeNullFields;
private final int _numSortFields;
private final RecordEnricherPipeline _recordEnricherPipeline;
- private final CompositeTransformer _recordTransformer;
- private final ComplexTypeTransformer _complexTypeTransformer;
+ private final TransformPipeline _transformPipeline;
private final TimeHandler _timeHandler;
private final Partitioner[] _partitioners;
private final String[] _partitionsBuffer;
// NOTE: Use TreeMap so that the order is deterministic
private final Map<String, GenericRowFileManager> _partitionToFileManagerMap
= new TreeMap<>();
- private AdaptiveSizeBasedWriter _adaptiveSizeBasedWriter;
- private List<RecordReaderFileConfig> _recordReaderFileConfigs;
- private List<RecordTransformer> _customRecordTransformers;
+ private final AdaptiveSizeBasedWriter _adaptiveSizeBasedWriter;
+ private final List<RecordReaderFileConfig> _recordReaderFileConfigs;
public SegmentMapper(List<RecordReaderFileConfig> recordReaderFileConfigs,
List<RecordTransformer> customRecordTransformers, SegmentProcessorConfig
processorConfig, File mapperOutputDir) {
+ this(recordReaderFileConfigs,
+ new TransformPipeline(
Review Comment:
Are we clear on how users will pass custom transformers? Up until now our
defaults were covering everything which hid this from users of
SegmentProcessorFramework. If we know for sure what defaults should be
applied, we should pass that here instead of having every SPF user to pass
that.
No-Op/empty list is one usecase but other than that Im not sure how users
will know what exactly to use. Which is better? Giving the option to
enable/disable the entire pipeline or allowing users to pass custom pipeline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
