shounakmk219 commented on code in PR #14425:
URL: https://github.com/apache/pinot/pull/14425#discussion_r1845852002
##########
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/mapper/SegmentMapper.java:
##########
@@ -71,21 +71,28 @@ public class SegmentMapper {
private final boolean _includeNullFields;
private final int _numSortFields;
private final RecordEnricherPipeline _recordEnricherPipeline;
- private final CompositeTransformer _recordTransformer;
- private final ComplexTypeTransformer _complexTypeTransformer;
+ private final TransformPipeline _transformPipeline;
private final TimeHandler _timeHandler;
private final Partitioner[] _partitioners;
private final String[] _partitionsBuffer;
// NOTE: Use TreeMap so that the order is deterministic
private final Map<String, GenericRowFileManager> _partitionToFileManagerMap
= new TreeMap<>();
- private AdaptiveSizeBasedWriter _adaptiveSizeBasedWriter;
- private List<RecordReaderFileConfig> _recordReaderFileConfigs;
- private List<RecordTransformer> _customRecordTransformers;
+ private final AdaptiveSizeBasedWriter _adaptiveSizeBasedWriter;
+ private final List<RecordReaderFileConfig> _recordReaderFileConfigs;
public SegmentMapper(List<RecordReaderFileConfig> recordReaderFileConfigs,
List<RecordTransformer> customRecordTransformers, SegmentProcessorConfig
processorConfig, File mapperOutputDir) {
+ this(recordReaderFileConfigs,
+ new TransformPipeline(
Review Comment:
We anyways have the simplified constructor which takes care of passing all
the default values as before so flows that don't want to tweak the default
behaviour can still use that. For flows that require some customisations like
for example let's say process segment data without applying the filters
(`FilterTransformer`), we need a way to pass a customised `transformPipeline`
as well hence the new constructor.
Right now we have `TransformPipeline.getPassThroughPipeline()` for the No-OP
usecase you mentioned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
