bobby-richard commented on code in PR #10045:
URL: https://github.com/apache/pinot/pull/10045#discussion_r1063467126
##########
pinot-connectors/pinot-flink-connector/src/main/java/org/apache/pinot/connector/flink/sink/PinotSinkFunction.java:
##########
@@ -151,4 +148,18 @@ private void flush()
LOG.info("Pinot segment uploaded to {}", segmentURI);
});
}
+
+ @Override
+ public List<GenericRow> snapshotState(long checkpointId, long timestamp) {
Review Comment:
Flink should manage all of the RocksDB complexity for us. Just clearing the
state at the appropriate times should be enough. I really think the state
solution is the way to go, otherwise the job will require very large heap
memory allocations.
Time based trigger for flush is interesting. I assume that would be
configurable? Storing the data in flink state for a very long time is really no
problem though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
