xingyingone opened a new issue, #31470: URL: https://github.com/apache/doris/issues/31470
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris-2.0.3-rc06-37d31a5 ### What's Wrong? I execute data load to doris and es the same time, but the data of doris is less than es, and I has execute retry logic in stream load code 1. select SQL and result SELECT COUNT(`@timestamp`) FROM eagle.envoy_log WHERE `@timestamp` >= '2024-02-26 08:00:00.0' AND `@timestamp` <= '2024-02-26 12:00:00.0'; es: 434,580,140 doris: 373,985,525 2. retry logic in stream load code ``` public void sendData(String table, String content, int retryTimes) { if (retryTimes <= 0) { log.error("stream load table:{}, content:{} failed ", table, content); throw new BizException("stream load failed", 500); } final String loadUrl = String.format("http://%s:%s/api/%s/%s/_stream_load";, DORIS_HOST[RandomUtils.nextInt(0, 3)], DORIS_HTTP_PORT, DORIS_DB, table); // log.info("doris sendData loadUrl:{}", loadUrl); HttpPut put = new HttpPut(loadUrl); put.removeHeaders(HttpHeaders.CONTENT_LENGTH); put.removeHeaders(HttpHeaders.TRANSFER_ENCODING); put.setHeader(HttpHeaders.EXPECT, "100-continue"); put.setHeader(HttpHeaders.AUTHORIZATION, DORIS_AUTHORIZATION); put.setHeader("format", "json"); put.setHeader("strip_outer_array", "true"); put.setHeader("load_to_single_tablet", "true"); // the label header is optional, not necessary // use label header can ensure at most once semantics put.setHeader("label", UUID.randomUUID().toString()); StringEntity entity = new StringEntity(content, "UTF-8"); put.setEntity(entity); try (CloseableHttpResponse response = client.execute(put)) { String loadResult = ""; if (response.getEntity() != null) { loadResult = EntityUtils.toString(response.getEntity()); } final int statusCode = response.getStatusLine().getStatusCode(); // statusCode 200 just indicates that doris be service is ok, not stream load // you should see the output content to find whether stream load is success if (logStreamLoadPrint) { log.info("Stream load table:{}, result: statusCode:{}, loadResult:{}", table, statusCode, loadResult); } if (statusCode != 200) { sendData(table, content, --retryTimes); } // httpcode == 200， status != ok StreamLoad streamLoad = JacksonUtil.from(loadResult, StreamLoad.class); if (streamLoad.getStatus().equalsIgnoreCase("Fail")) { sendData(table, content, --retryTimes); } log.info("Stream load table:{} success, retryTimes: {}", table, logStreamLoadTry - retryTimes); } } ``` ### What You Expected? no data loss ### How to Reproduce? I don‘ know ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
