RockteMQ-AI commented on issue #901: URL: https://github.com/apache/rocketmq-externals/issues/901#issuecomment-5250102174
**Issue Evaluation** Category: `type/question` This is a known behavior in Spark streaming with RocketMQ — when the application restarts, if the offset was not committed before the crash, the consumer will re-read from the last committed offset, causing repeated consumption. **Root Cause:** Spark Streaming commits offsets after processing. If the app crashes between consuming and committing, those messages are re-delivered on restart. This is by design for at-least-once semantics. **Mitigation:** - Ensure checkpointing is enabled (`checkpointDirectory`) so Spark can recover from the last checkpoint - Use `failedLimit` and `maxRetries` configurations to control retry behavior - For exactly-once semantics, implement idempotent consumers or use transactional processing --- *Automated evaluation by github-manager* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
