jackluo923 commented on code in PR #14534:
URL: https://github.com/apache/pinot/pull/14534#discussion_r1859076644
##########
pinot-plugins/pinot-input-format/pinot-clp-log/src/main/java/org/apache/pinot/plugin/inputformat/clplog/CLPLogRecordExtractorConfig.java:
##########
@@ -54,6 +55,7 @@ public class CLPLogRecordExtractorConfig implements
RecordExtractorConfig {
private final Set<String> _fieldsForClpEncoding = new HashSet<>();
private String _unencodableFieldSuffix = null;
private String _unencodableFieldError = null;
+ private boolean _removeProcessedFields = false;
Review Comment:
The default behavior is changed. Previously, field is removed by default,
now it's retained by default
##########
pinot-plugins/pinot-input-format/pinot-clp-log/src/main/java/org/apache/pinot/plugin/inputformat/clplog/CLPLogRecordExtractor.java:
##########
@@ -253,5 +253,9 @@ private void encodeFieldWithClp(String key, Object value,
GenericRow to) {
to.putValue(key + ClpRewriter.LOGTYPE_COLUMN_SUFFIX, logtype);
to.putValue(key + ClpRewriter.DICTIONARY_VARS_COLUMN_SUFFIX, dictVars);
to.putValue(key + ClpRewriter.ENCODED_VARS_COLUMN_SUFFIX, encodedVars);
+
+ if (!_config.getRemoveProcessedFields()) {
+ to.putValue(key, value);
+ }
Review Comment:
yes. if `removeProcessedField` is true, we won't add the original key and
value to the output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
