javrasya commented on issue #9444: URL: https://github.com/apache/iceberg/issues/9444#issuecomment-1886422859
This happens more often when consumption rate is high which is like replaying historical messages. When I run it in unbounded streaming mode and use `INCREMENTAL_FROM_EARLIEST_SNAPSHOT` streaming strategy instead of batch mode, the consumption rate drops inherently and this error occurs way less in a way that my app fails but recovers and continues and reaches to the end, very slowly but I will take it. Could this be happening because S3 is throttling or something, is there anyone else observed anything like this before? **Note:** The upstream is committing every minute which means that we are having new snapshot every minute which can also lead too many small files and this service which is having the respective error in the original post might be needing to pull too many files and eventually hitting that connection reset issue. This is just a theory, I couldn't verify it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
