sandeep-mst opened a new issue, #15: URL: https://github.com/apache/pulsar-connectors/issues/15
### Search before reporting - [x] I searched in the [issues](https://github.com/apache/pulsar-connectors/issues) and found nothing similar. ### Read release policy - [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### User environment master ### Issue Description The `KafkaConnectAdapter` currently lacks an effective backpressure mechanism to regulate message ingestion and buffering. As a result, it continues consuming records and appending them to the `pendingFlushQueue` even when downstream processing is saturated or stalled. At present, `batchSize` is only used as a flush threshold and does not impose any limit on writing records. This leads to accumulation of records in memory, which can result in OutOfMemoryError (OOM) under sustained load or when the sink is slow/unavailable. ### Error messages ```text ``` ### Reproducing the issue Run any sink using KafkaConnectAdapter (e.g., BigQuery sink) and simulate a slow or stalled downstream. Observe unbounded growth of `pendingFlushQueue` leading to OOM. ### Additional information _No response_ ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
