pnowojski opened a new pull request, #27589:
URL: https://github.com/apache/flink/pull/27589
This change depends on #27440
## What is the purpose of the change
Befor this change, when watermark alignment is enabled, it can prevent
backlogged jobs
from using all available resources. Inadvertently watermark alignment
configured with
maxAllowedWatermarkDrift and updateInterval was de facto capping the
backlog processing
speed to maxAllowedWatermarkDrift (event time) / updateInterval (wall
clock). For example
when maxAllowedWatermarkDrift=30s and updateInterval=1s, backlog could
not be processed
faster than 30s (event time) / 1s (wall clock). In that case, if job had
1 day of records
to process in the backlog (for example after 24h downtime), this backlog
could not be
processed more quickly than in 48 minutes, regardless of available
resources and number
of actual records.
This change adds SamplingWatermarksRingBuffer that will hide the latency
between
SourceOperators and SourceCoordinator. For more information please look
into the ticket.
## Brief change log
please check individual commits
## Verifying this change
PR adds new unit tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no**
/ don't know)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no ****/ don't
know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs
/ not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
