[
https://issues.apache.org/jira/browse/KAFKA-15063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omnia Ibrahim updated KAFKA-15063:
----------------------------------
Description:
{color:#172b4d}Ticket to track KIP-936. Since KIP-679 idempotent{color}
{color:#172b4d}producers became the default in Kafka as a result of this all
producer instances will be assigned PID. The increase of the number of PIDs
stored in Kafka brokers by {color}{{ProducerStateManager}}{color:#172b4d}
exposes the broker to OOM errors if it has a high number of producers, a rogue
or misconfigured client(s).{color}
{color:#172b4d}The broker is still exposed to OOM{color} even after KIP-854
introduced a separate config to expire PID from transaction IDs if there is a
high number of PID before {{producer.id.expiration.ms}} is exceeded.
As a result of this, the broker will keep experiencing OOM and become offline.
The only way to recover from this is to increase the heap.
{color:#172b4d}KIP-936 is proposing throttling the number of PIDs per
KafkaPrincipal {color}
{color:#172b4d}See the KIP details here
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs]
{color}
was:
{color:#172b4d}Ticket to track to track KIP-936. Since KIP-679
i{color:#172b4d}dempotent{color} {color:#172b4d}producers became the default in
Kafka {color:#172b4d}as a result of this all producer instances will be
assigned PID. The increase of number of PIDs stored in Kafka brokers by
{color}{{ProducerStateManager}}{color:#172b4d} exposes the broker to OOM errors
if it has a high number of producers, rogue or misconfigured client(s).{color}
{color}{color}
{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}The
broker is still exposed to OOM{color}{color}{color} even after KIP-854
introduced a separated config to expire PID from transaction IDs if there is
high number of PID before {color}{{producer.id.expiration.ms}}{color:#172b4d}
is exceeded.
{color}{color}{color}
As a result of this the broker will keep experincing OOM and become offline.
The only way to recover from this is to increase the heap.
{color:#172b4d}KIP-936 is proposing throttling number of PIDs per
KafkaPrincipal {color}
{color:#172b4d}See
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs]
{color}
> Throttle number of active PIDs
> ------------------------------
>
> Key: KAFKA-15063
> URL: https://issues.apache.org/jira/browse/KAFKA-15063
> Project: Kafka
> Issue Type: New Feature
> Components: core, producer
> Affects Versions: 2.8.0, 3.1.0, 3.0.0, 3.2.0, 3.3, 3.4.0
> Reporter: Omnia Ibrahim
> Priority: Major
>
> {color:#172b4d}Ticket to track KIP-936. Since KIP-679 idempotent{color}
> {color:#172b4d}producers became the default in Kafka as a result of this all
> producer instances will be assigned PID. The increase of the number of PIDs
> stored in Kafka brokers by {color}{{ProducerStateManager}}{color:#172b4d}
> exposes the broker to OOM errors if it has a high number of producers, a
> rogue or misconfigured client(s).{color}
> {color:#172b4d}The broker is still exposed to OOM{color} even after KIP-854
> introduced a separate config to expire PID from transaction IDs if there is a
> high number of PID before {{producer.id.expiration.ms}} is exceeded.
> As a result of this, the broker will keep experiencing OOM and become
> offline. The only way to recover from this is to increase the heap.
>
> {color:#172b4d}KIP-936 is proposing throttling the number of PIDs per
> KafkaPrincipal {color}
> {color:#172b4d}See the KIP details here
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs]
> {color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
