KKcorps commented on code in PR #9511:
URL: https://github.com/apache/pinot/pull/9511#discussion_r1008688878
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java:
##########
@@ -545,23 +545,31 @@ private boolean processStreamEvents(MessageBatch
messagesAndOffsets, long idlePi
// Decode message
StreamDataDecoderResult decodedRow =
_streamDataDecoder.decode(messagesAndOffsets.getStreamMessage(index));
RowMetadata msgMetadata =
messagesAndOffsets.getStreamMessage(index).getMetadata();
+ GenericRow decoderResult;
if (decodedRow.getException() != null) {
- // TODO: based on a config, decide whether the record should be
silently dropped or stop further consumption on
- // decode error
- realtimeRowsDroppedMeter =
- _serverMetrics.addMeteredTableValue(_metricKeyName,
ServerMeter.INVALID_REALTIME_ROWS_DROPPED, 1,
- realtimeRowsDroppedMeter);
+ if (_tableConfig.getIngestionConfig() != null
+ && _tableConfig.getIngestionConfig().isContinueOnError()) {
+ decoderResult = null;
+ realtimeRowsDroppedMeter =
+ _serverMetrics.addMeteredTableValue(_metricKeyName,
ServerMeter.INVALID_REALTIME_ROWS_DROPPED, 1,
+ realtimeRowsDroppedMeter);
+ } else {
+ throw new RuntimeException("Caught exception while decoding record",
decodedRow.getException());
Review Comment:
Marked PR as `backward-incompat` because of this line. I think some change
was done in 0.10 release because of which we were simply returning `null` here
and not throwing exception. However, this PR addresses a `TODO` present in that
code.
```java
// TODO: based on a config, decide whether the record should be silently
dropped or stop further consumption on
// decode error
```
After this change if `continueOnError` is not set in table config, we will
simply stop the consumption with an error. This can break old pipelines which
have been silently ignoring errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
