kumarpritam863 commented on code in PR #18739:
URL: https://github.com/apache/iceberg/pull/18739#discussion_r4520097621
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordProcessor.java:
##########
@@ -130,7 +149,31 @@ public void processElement(T element, Context ctx,
Collector<DynamicRecordIntern
@Override
public void collect(DynamicRecord inputData) {
DynamicRecordWithConfig data = dynamicRecordWithConfig.wrap(inputData);
+ try {
+ processRecord(data);
Review Comment:
Agreed, and it is visible end to end: on the forward path the NPE of the
chained writer ended up in the error stream and the job succeeded, whereas the
same record on the shuffle path failed the job. collect now only prepares the
record inside the try and emits it afterwards, so failures of chained operators
propagate. Tests: doesNotDivertFailureOfOperatorChainedToForwardOutput and
...ToMainOutput. They use a small hand-written Context/Collector, because the
harness cannot make a downstream operator fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
