TsukiokaKogane opened a new issue, #43241: URL: https://github.com/apache/doris/issues/43241
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 2.1.6 ### What's Wrong? in case of high throughput **zero error tolerance** stream load with csv format input, inaccurate response could occur like: ```json { "TxnId": 20035, "Label": "efe0c57c-ed07-4ee4-b61c-c2df93970fa1", "Comment": "", "TwoPhaseCommit": "false", "Status": "Fail", "Message": "[INTERNAL_ERROR]cancelled: closed", "NumberTotalRows": 0, "NumberLoadedRows": 0, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 0, "LoadTimeMs": 0, "BeginTxnTimeMs": 0, "StreamLoadPutTimeMs": 0, "ReadDataTimeMs": 0, "WriteDataTimeMs": 0, "CommitAndPublishTimeMs": 0, "ErrorURL": "" } ``` the reason behind that is because the csv reader detect the error in `CsvReader::_validate_line` and close the stream load pipe in `CsvReader::close` first. after that `StreamLoadAction::on_chunk_data` fails to append data to the pipe and set the stream load context status to <INTERNAL_ERROR] cancelled>. And finally in `StreamLoadAction::handle`, since the stream load context status is not ok, it just skips waiting and return the partial & inaccurate response before instance actually finished. ### What You Expected? stream load should return full response ### How to Reproduce? _No response_ ### Anything Else? I have 2 potential idea for fixing: one is to simply remove `if (ctx->status.ok()) ` condition in `StreamLoadAction::handle` ```cpp // status already set to fail if (ctx->status.ok()) { ctx->status = _handle(ctx); if (!ctx->status.ok() && !ctx->status.is<PUBLISH_TIMEOUT>()) { LOG(WARNING) << "handle streaming load failed, id=" << ctx->id << ", errmsg=" << ctx->status; } } ``` another is to set stream load status immediately after csv reader detects error Looking forward for more opinions! ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
