[
https://issues.apache.org/jira/browse/KAFKA-16641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848563#comment-17848563
]
Edoardo Comar commented on KAFKA-16641:
---------------------------------------
[~gharris] what pathological scenarios do you think of ?
frequent aborted transactions on a source topic?
source topic frequently deleted / recreated with same name ?
> MM2 offset translation should interpolate between sparse OffsetSyncs
> --------------------------------------------------------------------
>
> Key: KAFKA-16641
> URL: https://issues.apache.org/jira/browse/KAFKA-16641
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
> Reporter: Greg Harris
> Priority: Major
>
> Right now, the OffsetSyncStore keeps a sparse offset store, with exponential
> spacing between syncs. This can leave large gaps in translation, where
> offsets are translated much more conservatively than necessary.
> The dominant way to use MirrorMaker2 is in a "single writer" fashion, where
> the target topic is only written to by a single mirror maker 2. When a topic
> without gaps is replicated, contiguous blocks of offsets are preserved. For
> example:
> Say that MM2 mirrors 100 records, and emits two syncs: 0:100 and 100:200. We
> can detect when the gap between the upstream and downstream offsets is the
> same using subtraction, and then assume that 50:150 is also a valid
> translation. If the source topic has gaps, or goes through a restart, we
> should expect a discontinuity in the offset syncs, like 0:100 and 100:250 or
> 0:100 and 100:150.
> This may allow us to restore much of the offset translation precision that
> was lost for simple contiguous topics, without additional memory usage, but
> at the risk of mis-translating some pathological situations when the source
> topic has gaps. This might be able to be enabled unconditionally, or enabled
> via a configuration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
