[
https://issues.apache.org/jira/browse/KAFKA-16641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848631#comment-17848631
]
Adrian Preston edited comment on KAFKA-16641 at 5/23/24 11:38 AM:
------------------------------------------------------------------
-A further (related) question: assuming exactly once is enabled, can you
foresee situations when the downstream offset can advance without at least an
equal advance in the upstream offset?-
Actually, I think I've found the answer my own question :)
Enabling exactly once will result in transactions being used with the
downstream mirrored topic, so the markers will cause the downstream offset to
advance each time a transaction is committed / rolled back (without an equal
advance in the upstream offset).
was (Author: prestona):
A further (related) question: assuming exactly once is enabled, can you foresee
situations when the downstream offset can advance without at least an equal
advance in the upstream offset?
> MM2 offset translation should interpolate between sparse OffsetSyncs
> --------------------------------------------------------------------
>
> Key: KAFKA-16641
> URL: https://issues.apache.org/jira/browse/KAFKA-16641
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
> Reporter: Greg Harris
> Priority: Major
>
> Right now, the OffsetSyncStore keeps a sparse offset store, with exponential
> spacing between syncs. This can leave large gaps in translation, where
> offsets are translated much more conservatively than necessary.
> The dominant way to use MirrorMaker2 is in a "single writer" fashion, where
> the target topic is only written to by a single mirror maker 2. When a topic
> without gaps is replicated, contiguous blocks of offsets are preserved. For
> example:
> Say that MM2 mirrors 100 records, and emits two syncs: 0:100 and 100:200. We
> can detect when the gap between the upstream and downstream offsets is the
> same using subtraction, and then assume that 50:150 is also a valid
> translation. If the source topic has gaps, or goes through a restart, we
> should expect a discontinuity in the offset syncs, like 0:100 and 100:250 or
> 0:100 and 100:150.
> This may allow us to restore much of the offset translation precision that
> was lost for simple contiguous topics, without additional memory usage, but
> at the risk of mis-translating some pathological situations when the source
> topic has gaps. This might be able to be enabled unconditionally, or enabled
> via a configuration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
