aishikbh commented on code in PR #12220:
URL: https://github.com/apache/pinot/pull/12220#discussion_r1449310943
##########
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/mapper/SegmentMapper.java:
##########
@@ -141,28 +146,43 @@ private Map<String, GenericRowFileManager> doMap()
RecordReaderFactory.getRecordReader(recordReaderFileConfig._fileFormat,
recordReaderFileConfig._dataFile,
recordReaderFileConfig._fieldsToRead,
recordReaderFileConfig._recordReaderConfig);
mapAndTransformRow(recordReader, reuse, observer, count, totalCount);
+ _recordReaderFileConfigs.get(i)._recordReader = recordReader;
} finally {
- if (recordReader != null) {
+ if (recordReader != null && !recordReader.hasNext()) {
recordReader.close();
}
}
} else {
+ if (!recordReader.hasNext()) {
+ LOGGER.info("Skipping record reader as it is already processed at
index: {}", i);
+ count++;
+ continue;
+ }
mapAndTransformRow(recordReader, reuse, observer, count, totalCount);
+ _recordReaderFileConfigs.get(i)._recordReader = recordReader;
+ }
+ if (!_adaptiveSizeBasedWriter.canWrite()) {
Review Comment:
On further inspection I have kept the check only in one place. Right after
`mapAndTransformRow`. If we got kicked out of `mapAndTransformRow` because of
size constraints this check should be enough.
Please let me know if returning a boolean value makes more sense. I will
change accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
