gnehil opened a new pull request, #148: URL: https://github.com/apache/doris-spark-connector/pull/148
# Problem Summary In the current implementation, due to traversing data through iterators and sending data in a stream, batch loading does not implement batch retry. Task retry will be performed after the load request fails, and the cost of task retry is relatively high. For some scenarios such as network jitter, batch retry is a better way ## Proposed changes: The implementation of this PR is to choose whether to retry batch loading when it fails through configuration. The behavior is as follows： Batch retry is enabled when sink.max-retries is set greater than 0，an iterator is implemented to cache batch data, and a count value is used to control batch writing. When the batch fails, reset the count and set the flag bit, and re-read the current batch data from the cache array. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
