date:20240707

[PR] [fix] fix date hardcode to date function [doris]

2024-07-07 Thread via GitHub



133tosakarin opened a new pull request, #37389:
URL: https://github.com/apache/doris/pull/37389

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix] fix date hardcode to date function [doris]

2024-07-07 Thread via GitHub



133tosakarin commented on PR #37389:
URL: https://github.com/apache/doris/pull/37389#issuecomment-2212347093

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [enhancement](regression-test) dup modify key [doris]

2024-07-07 Thread via GitHub



cjj2010 opened a new pull request, #37390:
URL: https://github.com/apache/doris/pull/37390

   ## Proposed changes
   
   Issue Number: close #xxx
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [enhancement](regression-test) dup modify key [doris]

2024-07-07 Thread via GitHub



cjj2010 commented on PR #37390:
URL: https://github.com/apache/doris/pull/37390#issuecomment-2212347409

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [branch-2.1](routine-load) optimize out of range error message [doris]

2024-07-07 Thread via GitHub



sollhui opened a new pull request, #37391:
URL: https://github.com/apache/doris/pull/37391

   ## Proposed changes
   pick #36450
   
   before
   ```
   ErrorReason{code=errCode = 105, msg='be 10002 abort task, task id: 
d846f3d3-7c9e-44a7-bee0-3eff8cd11c6f job id: 11310 with reason: 
[INTERNAL_ERROR]Offset out of range,
   
   0#  doris::Status doris::Status::Error<6, 
true>(std::basic_string_view >) at 
/mnt/disk1/laihui/doris/be/src/common/status.h:422
   1#  doris::Status 
doris::Status::InternalError(std::basic_string_view >) at /mnt/disk1/laihui/doris/be/src/common/status.h:468
   2#  
doris::KafkaDataConsumer::group_consume(doris::BlockingQueue*,
 long) at 
/mnt/disk1/laihui/doris/be/src/runtime/routine_load/data_consumer.cpp:226
   3#  
doris::KafkaDataConsumerGroup::actual_consume(std::shared_ptr,
 doris::BlockingQueue*, long, std::function) at 
/mnt/disk1/laihui/doris/be/src/runtime/routine_load/data_consumer_group.cpp:200
   4#  void std::__invoke_impl, 
doris::BlockingQueue*, long, std::function), doris::KafkaDataConsumerGroup*&, 
std::shared_ptr&, 
doris::BlockingQueue*&, long&, 
doris::KafkaDataConsumerGroup::start_all(std::shared_ptr,
 
std::shared_ptr)::$_0&>(std::__invoke_memfun_deref,
 void (doris::KafkaDataConsumerGroup::*&)(std::shared_ptr, 
doris::BlockingQueue*, long, std::function), doris::KafkaDataConsumerGroup*&, 
std::shared_ptr&, 
doris::BlockingQueue*&, long&, 
doris::KafkaDataConsumerGroup::start_all(std::shared_ptr,
 std::shared_ptr)::$_0&) at 
/mnt/disk1/laihui/build/ldb_toolchain/bin/../lib/gcc/x86_6
 4-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74
   ...
   ```
   
   now
   ```
   ErrorReason{code=errCode = 105, msg='be 10002 abort task, task id: 
3ba0c0f4-d13c-4dfa-90ce-3df922fd9340 job id: 11310 with reason: 
[INTERNAL_ERROR]Offset out of range, consume partition 0, consume offset 100, 
the offset used by job does not exist in kafka, please check the offset, using 
the Alter ROUTINE LOAD command to modify it, and resume the job'}
   ```
   
   ## Proposed changes
   
   Issue Number: close #xxx
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [branch-2.1](routine-load) optimize out of range error message [doris]

2024-07-07 Thread via GitHub



sollhui commented on PR #37391:
URL: https://github.com/apache/doris/pull/37391#issuecomment-2212347461

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [branch-2.1](routine-load) optimize out of range error message [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37391:
URL: https://github.com/apache/doris/pull/37391#issuecomment-2212348913

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [opt](serde)Optimize the filling of fixed values into block columns without repeated deserialization. [doris]

2024-07-07 Thread via GitHub



hubgeter commented on PR #37377:
URL: https://github.com/apache/doris/pull/37377#issuecomment-2212354319

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [opt](serde)Optimize the filling of fixed values into block columns without repeated deserialization. [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37377:
URL: https://github.com/apache/doris/pull/37377#issuecomment-2212355606

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [regression](cloud) Fix missing hadoop.username when create hdfs vault [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37381:
URL: https://github.com/apache/doris/pull/37381#issuecomment-2212356085

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [regression](cloud) Fix missing hadoop.username when create hdfs vault [doris]

2024-07-07 Thread via GitHub



gavinchou merged PR #37381:
URL: https://github.com/apache/doris/pull/37381


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris) branch master updated (c111539b715 -> baba531b78a)

2024-07-07 Thread gavinchou

This is an automated email from the ASF dual-hosted git repository.

gavinchou pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


from c111539b715 [enhancement](regression-test) dup modify value case3 
(#37359)
 add baba531b78a [regression](cloud) Fix missing hadoop.username when 
create hdfs vault (#37381)

No new revisions were added by this update.

Summary of changes:
 .../suites/hdfs_vault_p2/default_vault_p2/load.groovy |  6 --
 regression-test/suites/vaults/create/create.groovy| 11 +++
 regression-test/suites/vaults/default/default.groovy  |  5 +++--
 3 files changed, 14 insertions(+), 8 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [opt](serde)Optimize the filling of fixed values into block columns without repeated deserialization. [doris]

2024-07-07 Thread via GitHub



hubgeter commented on PR #37377:
URL: https://github.com/apache/doris/pull/37377#issuecomment-2212357941

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [opt](serde)Optimize the filling of fixed values into block columns without repeated deserialization. [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37377:
URL: https://github.com/apache/doris/pull/37377#issuecomment-2212359094

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [enhancement](regression-test) dup modify key [doris]

2024-07-07 Thread via GitHub



cjj2010 commented on PR #37390:
URL: https://github.com/apache/doris/pull/37390#issuecomment-2212369575

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [enhancement](cloud) support TTL file cache evict through LRU [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37312:
URL: https://github.com/apache/doris/pull/37312#issuecomment-2212376947

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [chore](memory) ScopedAllocatorTagger [doris]

2024-07-07 Thread via GitHub



zhiqiang- commented on PR #37066:
URL: https://github.com/apache/doris/pull/37066#issuecomment-2212383673

   run p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[I] [Feature] 支持sftp或ftp协议，导入导出数据。 [doris]

2024-07-07 Thread via GitHub



wanezu opened a new issue, #37392:
URL: https://github.com/apache/doris/issues/37392

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no 
similar issues.
   
   
   ### Description
   
   是否可以支持ftp和sftp导入和导出数据，对于一些从传统数据库到doris转型的用户，独立搭建s3或hdfs，成本还是挺高的，包括后期的维护成本。
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](function) Fix wrong result of week/yearweek in constant folding [doris]

2024-07-07 Thread via GitHub



zclllyybb commented on PR #37376:
URL: https://github.com/apache/doris/pull/37376#issuecomment-2212389941

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix] forget commit .out file [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37387:
URL: https://github.com/apache/doris/pull/37387#issuecomment-2212397420

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix] forget commit .out file [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37387:
URL: https://github.com/apache/doris/pull/37387#issuecomment-2212397437

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [branch-2.1](routine-load) add retry when get Kafka meta info [doris]

2024-07-07 Thread via GitHub



dataroaring merged PR #37371:
URL: https://github.com/apache/doris/pull/37371


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris) branch branch-2.1 updated: [branch-2.1](routine-load) add retry when get Kafka meta info (#37371)

2024-07-07 Thread dataroaring

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
 new 960b02f2939 [branch-2.1](routine-load) add retry when get Kafka meta 
info (#37371)
960b02f2939 is described below

commit 960b02f29392a6bd1afc1b682fc39d758f5f58bc
Author: hui lai <1353307...@qq.com>
AuthorDate: Sun Jul 7 18:14:38 2024 +0800

[branch-2.1](routine-load) add retry when get Kafka meta info (#37371)

pick #35376
---
 .../apache/doris/datasource/kafka/KafkaUtil.java   | 171 -
 1 file changed, 66 insertions(+), 105 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/datasource/kafka/KafkaUtil.java 
b/fe/fe-core/src/main/java/org/apache/doris/datasource/kafka/KafkaUtil.java
index 00169f4c8ea..656ebf65152 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/kafka/KafkaUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/kafka/KafkaUtil.java
@@ -43,24 +43,11 @@ import java.util.stream.Collectors;
 public class KafkaUtil {
 private static final Logger LOG = LogManager.getLogger(KafkaUtil.class);
 private static final int MAX_KAFKA_PARTITION_TIMEOUT_SECOND = 60;
-private static final int MAX_GET_OFFSET_TIMEOUT_SECOND = 5;
+private static final int MAX_GET_OFFSET_TIMEOUT_SECOND = 10;
 
 public static List getAllKafkaPartitions(String brokerList, 
String topic,
 Map convertedCustomProperties) throws 
UserException {
-TNetworkAddress address = null;
-Backend be = null;
-long beId = -1L;
 try {
-List backendIds = 
Env.getCurrentSystemInfo().getAllBackendIds(true);
-if (backendIds.isEmpty()) {
-throw new LoadException("Failed to get all partitions. No 
alive backends");
-}
-Collections.shuffle(backendIds);
-beId = backendIds.get(0);
-be = Env.getCurrentSystemInfo().getBackend(beId);
-address = new TNetworkAddress(be.getHost(), be.getBrpcPort());
-
-// create request
 InternalService.PProxyRequest request = 
InternalService.PProxyRequest.newBuilder().setKafkaMetaRequest(
 InternalService.PKafkaMetaProxyRequest.newBuilder()
 
.setKafkaInfo(InternalService.PKafkaLoadInfo.newBuilder()
@@ -72,21 +59,10 @@ public class KafkaUtil {
 )
 )
 ).build();
-
-// get info
-Future future = 
BackendServiceProxy.getInstance().getInfo(address, request);
-InternalService.PProxyResult result = 
future.get(MAX_KAFKA_PARTITION_TIMEOUT_SECOND, TimeUnit.SECONDS);
-TStatusCode code = 
TStatusCode.findByValue(result.getStatus().getStatusCode());
-if (code != TStatusCode.OK) {
-throw new UserException("failed to get kafka partition info: " 
+ result.getStatus().getErrorMsgsList());
-} else {
-return result.getKafkaMetaResult().getPartitionIdsList();
-}
+return getInfoRequest(request, 
MAX_GET_OFFSET_TIMEOUT_SECOND).getKafkaMetaResult().getPartitionIdsList();
 } catch (Exception e) {
-LOG.warn("failed to get partitions from backend[{}].", beId, e);
 throw new LoadException(
-"Failed to get all partitions of kafka topic: " + topic + 
" from backend[" + beId
-+ "]. error: " + e.getMessage());
+"Failed to get all partitions of kafka topic: " + topic + 
" error: " + e.getMessage());
 }
 }
 
@@ -96,20 +72,10 @@ public class KafkaUtil {
 public static List> getOffsetsForTimes(String 
brokerList, String topic,
 Map convertedCustomProperties, List> timestampOffsets)
 throws LoadException {
-TNetworkAddress address = null;
 if (LOG.isDebugEnabled()) {
 LOG.debug("begin to get offsets for times of topic: {}, {}", 
topic, timestampOffsets);
 }
 try {
-List backendIds = 
Env.getCurrentSystemInfo().getAllBackendIds(true);
-if (backendIds.isEmpty()) {
-throw new LoadException("Failed to get offset for times. No 
alive backends");
-}
-Collections.shuffle(backendIds);
-Backend be = 
Env.getCurrentSystemInfo().getBackend(backendIds.get(0));
-address = new TNetworkAddress(be.getHost(), be.getBrpcPort());
-
-// create request
 InternalService.PKafkaMetaProxyRequest.Builder metaRequestBuilder =
 InternalService.PKafkaMetaProxyRequest.newBuilder()
 
.setKafkaInfo(InternalService.PKafkaLoadInfo.newBuilder()
@@ -131,24 +97,17 @@ public class KafkaUtil {

Re: [PR] [branch-2.1](routine-load) dealing with the high watermark of Kafka may fallback [doris]

2024-07-07 Thread via GitHub



dataroaring merged PR #37372:
URL: https://github.com/apache/doris/pull/37372


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris) branch branch-2.1 updated: [branch-2.1](routine-load) dealing with the high watermark of Kafka may fallback (#37372)

2024-07-07 Thread dataroaring

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
 new af960f7c70f [branch-2.1](routine-load) dealing with the high watermark 
of Kafka may fallback (#37372)
af960f7c70f is described below

commit af960f7c70f8569d856b30be36515417bf8dbc67
Author: hui lai <1353307...@qq.com>
AuthorDate: Sun Jul 7 18:15:54 2024 +0800

[branch-2.1](routine-load) dealing with the high watermark of Kafka may 
fallback (#37372)

pick #35901
---
 .../org/apache/doris/load/routineload/KafkaRoutineLoadJob.java   | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java
 
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java
index 43ae98d8f7a..201412027ab 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java
@@ -765,7 +765,14 @@ public class KafkaRoutineLoadJob extends RoutineLoadJob {
 List> tmp = KafkaUtil.getLatestOffsets(id, 
taskId, getBrokerList(),
 getTopic(), getConvertedCustomProperties(), 
Lists.newArrayList(partitionIdToOffset.keySet()));
 for (Pair pair : tmp) {
-cachedPartitionWithLatestOffsets.put(pair.first, pair.second);
+if (pair.second >= 
cachedPartitionWithLatestOffsets.getOrDefault(pair.first, Long.MIN_VALUE)) {
+cachedPartitionWithLatestOffsets.put(pair.first, 
pair.second);
+} else {
+LOG.warn("Kafka offset fallback. partition: {}, cache 
offset: {}"
++ " get latest offset: {}, task {}, job {}",
+pair.first, 
cachedPartitionWithLatestOffsets.getOrDefault(pair.first, Long.MIN_VALUE),
+pair.second, taskId, id);
+}
 }
 } catch (Exception e) {
 // It needs to pause job when can not get partition meta.


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [branch-2.1](routine-load) optimize routine load job auto resume policy [doris]

2024-07-07 Thread via GitHub



dataroaring merged PR #37373:
URL: https://github.com/apache/doris/pull/37373


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris) branch branch-2.1 updated: [branch-2.1](routine-load) optimize routine load job auto resume policy (#37373)

2024-07-07 Thread dataroaring

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
 new d08a418dd85 [branch-2.1](routine-load) optimize routine load job auto 
resume policy (#37373)
d08a418dd85 is described below

commit d08a418dd85d6eb5f01aecd1f5aaa7568c781ef1
Author: hui lai <1353307...@qq.com>
AuthorDate: Sun Jul 7 18:16:56 2024 +0800

[branch-2.1](routine-load) optimize routine load job auto resume policy 
(#37373)

pick #35266
---
 .../main/java/org/apache/doris/common/Config.java  |  2 +-
 .../doris/load/routineload/RoutineLoadJob.java |  3 +-
 .../doris/load/routineload/RoutineLoadManager.java |  3 +-
 .../doris/load/routineload/ScheduleRule.java   | 40 ++
 .../load/routineload/RoutineLoadManagerTest.java   | 11 +++---
 5 files changed, 27 insertions(+), 32 deletions(-)

diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java 
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 26ef8852b45..94d5725c38a 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -1257,7 +1257,7 @@ public class Config extends ConfigBase {
  * a period for auto resume routine load
  */
 @ConfField(mutable = true, masterOnly = true)
-public static int period_of_auto_resume_min = 5;
+public static int period_of_auto_resume_min = 10;
 
 /**
  * If set to true, the backend will be automatically dropped after 
finishing decommission.
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
 
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
index 8c8dd7eaad9..130bd87b018 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
@@ -224,9 +224,8 @@ public abstract class RoutineLoadJob extends 
AbstractTxnStateChangeCallback impl
 protected int currentTaskConcurrentNum;
 protected RoutineLoadProgress progress;
 
-protected long firstResumeTimestamp; // the first resume time
+protected long latestResumeTimestamp; // the latest resume time
 protected long autoResumeCount;
-protected boolean autoResumeLock = false; //it can't auto resume iff true
 // some other msg which need to show to user;
 protected String otherMsg = "";
 protected ErrorReason pauseReason;
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java
 
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java
index 356262f8c2a..6121f34035c 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java
@@ -367,8 +367,7 @@ public class RoutineLoadManager implements Writable {
 try {
 routineLoadJob.jobStatistic.errorRowsAfterResumed = 0;
 routineLoadJob.autoResumeCount = 0;
-routineLoadJob.firstResumeTimestamp = 0;
-routineLoadJob.autoResumeLock = false;
+routineLoadJob.latestResumeTimestamp = 0;
 
routineLoadJob.updateState(RoutineLoadJob.JobState.NEED_SCHEDULE, null, false 
/* not replay */);
 LOG.info(new LogBuilder(LogKey.ROUTINE_LOAD_JOB, 
routineLoadJob.getId())
 .add("current_state", routineLoadJob.getState())
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/ScheduleRule.java 
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/ScheduleRule.java
index 2b4ef7b297a..dfc47aa496e 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/ScheduleRule.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/ScheduleRule.java
@@ -31,6 +31,10 @@ import org.apache.logging.log4j.Logger;
 public class ScheduleRule {
 private static final Logger LOG = LogManager.getLogger(ScheduleRule.class);
 
+private static final long BACK_OFF_BASIC_TIME_SEC = 10L;
+
+private static final long MAX_BACK_OFF_TIME_SEC = 60 * 5;
+
 private static int deadBeCount() {
 SystemInfoService systemInfoService = Env.getCurrentSystemInfo();
 int total = systemInfoService.getAllBackendIds(false).size();
@@ -47,22 +51,10 @@ public class ScheduleRule {
 if (jobRoutine.state != RoutineLoadJob.JobState.PAUSED) {
 return false;
 }
-if (jobRoutine.autoResumeLock) { //only manual resume for unlock
-if (LOG.isDebugEnabled()) {
-LOG.debug("routine load job {}'s autoResumeLock is true, 
skip", jobRoutine.id);
-}
-

Re: [PR] [fix](txn insert) sub txn map is not thread safe [doris]

2024-07-07 Thread via GitHub



dataroaring merged PR #37265:
URL: https://github.com/apache/doris/pull/37265


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris) branch master updated: [fix](txn insert) sub txn map is not thread safe (#37265)

2024-07-07 Thread dataroaring

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 8e365e873dc [fix](txn insert) sub txn map is not thread safe (#37265)
8e365e873dc is described below

commit 8e365e873dc565bc85e132a06e54ad3a93214bca
Author: meiyi 
AuthorDate: Sun Jul 7 18:18:07 2024 +0800

[fix](txn insert) sub txn map is not thread safe (#37265)

## Proposed changes

The sub txn map in DatabaseTransactionMgr is not thread safe
---
 .../main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
 
b/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
index df9802f5eaa..cd2ae903986 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
@@ -135,7 +135,7 @@ public class DatabaseTransactionMgr {
 
 // transactionId -> final status TransactionState
 private final Map idToFinalStatusTransactionState 
= Maps.newHashMap();
-private final Map subTxnIdToTxnId = Maps.newHashMap();
+private final Map subTxnIdToTxnId = new ConcurrentHashMap<>();
 
 // The following 2 queues are to store transactionStates with final status
 // These queues are mainly used to avoid traversing all txns and speed up 
the cleaning time


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](multi-table-load) fix be core when multi table load pipe finish fail [doris]

2024-07-07 Thread via GitHub



dataroaring merged PR #37383:
URL: https://github.com/apache/doris/pull/37383


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris) branch branch-2.1 updated: [fix](multi-table-load) fix be core when multi table load pipe finish fail (#37383)

2024-07-07 Thread dataroaring

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
 new f2693152bb9 [fix](multi-table-load) fix be core when multi table load 
pipe finish fail (#37383)
f2693152bb9 is described below

commit f2693152bb96ab24148a50f858621c692b2533e3
Author: hui lai <1353307...@qq.com>
AuthorDate: Sun Jul 7 18:24:16 2024 +0800

[fix](multi-table-load) fix be core when multi table load pipe finish fail 
(#37383)

pick #36269
---
 be/src/runtime/routine_load/routine_load_task_executor.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/be/src/runtime/routine_load/routine_load_task_executor.cpp 
b/be/src/runtime/routine_load/routine_load_task_executor.cpp
index 137a2c37f68..9b475ed2133 100644
--- a/be/src/runtime/routine_load/routine_load_task_executor.cpp
+++ b/be/src/runtime/routine_load/routine_load_task_executor.cpp
@@ -402,7 +402,7 @@ void 
RoutineLoadTaskExecutor::exec_task(std::shared_ptr ctx,
 }
 // need memory order
 multi_table_pipe->handle_consume_finished();
-HANDLE_ERROR(kafka_pipe->finish(), "finish multi table task failed");
+HANDLE_MULTI_TABLE_ERROR(kafka_pipe->finish(), "finish multi table 
task failed");
 } else {
 // start to consume, this may block a while
 HANDLE_ERROR(consumer_grp->start_all(ctx, kafka_pipe), "consuming 
failed");


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [branch-2.1](move-memtable) fix move memtable core when use multi table load [doris]

2024-07-07 Thread via GitHub



dataroaring merged PR #37370:
URL: https://github.com/apache/doris/pull/37370


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris) branch branch-2.1 updated (f2693152bb9 -> 61bc624938b)

2024-07-07 Thread dataroaring

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a change to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


from f2693152bb9 [fix](multi-table-load) fix be core when multi table load 
pipe finish fail (#37383)
 add 61bc624938b [branch-2.1](move-memtable) fix move memtable core when 
use multi table load (#37370)

No new revisions were added by this update.

Summary of changes:
 be/src/io/file_factory.cpp|  20 
 be/src/io/fs/multi_table_pipe.cpp | 216 +-
 be/src/io/fs/multi_table_pipe.h   |  11 +-
 3 files changed, 98 insertions(+), 149 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris) branch branch-2.1 updated: [cherry-pick](branch-2.1) Pick "[Enhancement](partial update) Add partial update mix cases (#37113)" (#37384)

2024-07-07 Thread dataroaring

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
 new 32529ecda27 [cherry-pick](branch-2.1) Pick "[Enhancement](partial 
update) Add partial update mix cases (#37113)" (#37384)
32529ecda27 is described below

commit 32529ecda2754780efcc8acf3252e9ba94341510
Author: abmdocrt 
AuthorDate: Sun Jul 7 18:26:46 2024 +0800

[cherry-pick](branch-2.1) Pick "[Enhancement](partial update) Add partial 
update mix cases (#37113)" (#37384)

#37113
---
 .../partial_update/test_mix_partial_update.out | 365 
 .../test_mix_partial_update_load1.csv  |   4 +
 .../test_mix_partial_update_load2.csv  |   1 +
 .../test_mix_partial_update_load3.csv  |   1 +
 .../test_mix_partial_update_load4.csv  |   1 +
 .../test_mix_partial_update_load5.csv  |   1 +
 .../test_mix_partial_update_load6.csv  |   1 +
 .../test_mix_partial_update_load7.csv  |   1 +
 .../test_mix_partial_update_load_A.csv |   1 +
 .../test_mix_partial_update_load_B.csv |   1 +
 .../test_mix_partial_update_load_C.csv |   1 +
 .../test_mix_partial_update_load_D.csv |   1 +
 .../partial_update/test_mix_partial_update.groovy  | 381 +
 13 files changed, 760 insertions(+)

diff --git 
a/regression-test/data/unique_with_mow_p0/partial_update/test_mix_partial_update.out
 
b/regression-test/data/unique_with_mow_p0/partial_update/test_mix_partial_update.out
new file mode 100644
index 000..13b699ae13c
--- /dev/null
+++ 
b/regression-test/data/unique_with_mow_p0/partial_update/test_mix_partial_update.out
@@ -0,0 +1,365 @@
+-- This file is automatically generated. You should know what you did if you 
want to edit this
+-- !select1 --
+1  1   1   1   10
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select2 --
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select3 --
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select4 --
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select5 --
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select6 --
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select7 --
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select8 --
+2  2   2   2   2
+3  3   3   3   3
+4  4   4   4   4
+
+-- !select9 --
+2  2   2   2   2
+3  3   3   3   3
+4  4   4   4   4
+
+-- !select1 --
+1  1   1   1   10
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select2 --
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select3 --
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select4 --
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select5 --
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select6 --
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select7 --
+2  2   2   2   2
+3  3   3   3   3
+
+-- !select8 --
+2  2   2   2   2
+3  3   3   3   3
+4  4   4   4   4
+
+-- !select9 --
+2  2   2   2   2
+3  3   3   3   3
+4  4   4   4   4
+
+-- !select_A --
+1  1   1   {"a":100, "b":100}
+
+-- !select_AA --
+1
+
+-- !select_AAA --
+1
+
+-- !select_B --
+1  1   1   {"a":100, "b":100}
+2  2   2   {"a":200, "b":200}
+
+-- !select_BB --
+2
+
+-- !select_BBB --
+2
+
+-- !select_C --
+1  1   1   {"a":100, "b":100}
+2  2   2   {"a":200, "b":200}
+
+-- !select_CC --
+1
+
+-- !select_CCC --
+2
+
+-- !select_D --
+1  1   1   {"a":100, "b":100}
+2  2   2   {"a":200, "b":200}
+3  3   3   {"a":300, "b":300}
+
+-- !select_DD --
+2
+
+-- !select_DDD --
+1
+
+-- !select_E --
+1  1   1   {"a":100, "b":100}
+2  2   2   {"a":200, "b":200}
+3  3   3   {"a":300, "b":300}
+4  4   4   {"a":400, "b":400}
+
+-- !select_EE --
+3
+
+-- !select_EEE --
+2
+
+-- !select_A --
+1  1   1   {"a":100, "b":100}
+
+-- !select_AA --
+1
+
+-- !select_AAA --
+1
+
+-- !select_B --
+1  1   1   {"a":100, "b":100}
+2  2   2   {"a":200, "b":200}
+
+-- !select_BB --
+2
+
+-- !select_BBB --
+2
+
+-- !select_C --
+1  1   1   {"a":100, "b":100}
+2  2   2   {"a":200, "b":200}
+
+-- !select_CC --
+1
+
+-- !select_CCC --
+2
+
+-- !select_

Re: [PR] [cherry-pick](branch-2.1) Pick "[Enhancement](partial update) Add partial update mix cases (#37113)" [doris]

2024-07-07 Thread via GitHub



dataroaring merged PR #37384:
URL: https://github.com/apache/doris/pull/37384


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [chery-pick](branch-2.1) Pick "[Fix](group commit) Fix group commit block queue mem estimate fault" [doris]

2024-07-07 Thread via GitHub



dataroaring merged PR #37379:
URL: https://github.com/apache/doris/pull/37379


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris) branch branch-2.1 updated: [chery-pick](branch-2.1) Pick "[Fix](group commit) Fix group commit block queue mem estimate fault" (#37379)

2024-07-07 Thread dataroaring

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
 new 7d423b3a6a2 [chery-pick](branch-2.1) Pick "[Fix](group commit) Fix 
group commit block queue mem estimate fault" (#37379)
7d423b3a6a2 is described below

commit 7d423b3a6a2ffaba18584f8400795ef56838114f
Author: abmdocrt 
AuthorDate: Sun Jul 7 18:27:49 2024 +0800

[chery-pick](branch-2.1) Pick "[Fix](group commit) Fix group commit block 
queue mem estimate fault" (#37379)

Pick [Fix](group commit) Fix group commit block queue mem estimate faule
#35314

## Proposed changes

Issue Number: close #xxx



**Problem:** When `group commit=async_mode` and NULL data is imported
into a `variant` type column, it causes incorrect memory statistics for
group commit backpressure, leading to a stuck issue. **Cause:** In group
commit mode, blocks are first added to a queue in batches using `add
block`, and then blocks are retrieved from the queue using `get block`.
To track memory usage during backpressure, we add the block size to the
memory statistics during `add block` and subtract the block size from
the memory statistics during `get block`. However, for `variant` types,
during the `add block` write to WAL, serialization occurs, which can
merge types (e.g., merging `int` and `bigint` into `bigint`), thereby
changing the block size. This results in a discrepancy between the block
size during `get block` and `add block`, causing memory statistics to
overflow.
**Solution:** Record the block size at the time of `add block` and use
this recorded size during `get block` instead of the actual block size.
This ensures consistency in the memory addition and subtraction.

## Further comments

If this is a relatively large or complex change, kick off the discussion
at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why
you chose the solution you did and what alternatives you considered,
etc...

## Proposed changes

Issue Number: close #xxx


---
 be/src/runtime/group_commit_mgr.cpp| 66 +++---
 be/src/runtime/group_commit_mgr.h  |  9 ++-
 be/src/vec/core/block.cpp  | 19 +++
 be/src/vec/core/block.h|  2 +
 .../insert_p0/test_group_commit_variant.groovy | 43 ++
 5 files changed, 131 insertions(+), 8 deletions(-)

diff --git a/be/src/runtime/group_commit_mgr.cpp 
b/be/src/runtime/group_commit_mgr.cpp
index 38e599180e7..d5daf2af530 100644
--- a/be/src/runtime/group_commit_mgr.cpp
+++ b/be/src/runtime/group_commit_mgr.cpp
@@ -62,9 +62,27 @@ Status LoadBlockQueue::add_block(RuntimeState* runtime_state,
 RETURN_IF_ERROR(status);
 if (block->rows() > 0) {
 if (!config::group_commit_wait_replay_wal_finish) {
-_block_queue.push_back(block);
+_block_queue.emplace_back(block);
 _data_bytes += block->bytes();
+int before_block_queues_bytes = _all_block_queues_bytes->load();
 _all_block_queues_bytes->fetch_add(block->bytes(), 
std::memory_order_relaxed);
+std::stringstream ss;
+ss << "[";
+for (const auto& id : _load_ids) {
+ss << id.to_string() << ", ";
+}
+ss << "]";
+VLOG_DEBUG << "[Group Commit Debug] (LoadBlockQueue::add_block). "
+   << "block queue size is " << _block_queue.size() << ", 
block rows is "
+   << block->rows() << ", block bytes is " << 
block->bytes()
+   << ", before add block, all block queues bytes is "
+   << before_block_queues_bytes
+   << ", after add block, all block queues bytes is "
+   << _all_block_queues_bytes->load() << ", txn_id=" << 
txn_id
+   << ", label=" << label << ", instance_id=" << 
load_instance_id
+   << ", load_ids=" << ss.str() << ", runtime_state=" << 
runtime_state
+   << ", the block is " << block->dump_data() << ", the 
block column size is "
+   << block->columns_bytes();
 }
 if (write_wal || config::group_commit_wait_replay_wal_finish) {
 auto st = _v_wal_writer->write_wal(block.get());
@@ -132,11 +150,29 @@ Status LoadBlockQueue::get_block(RuntimeState* 
runtime_state, vectorized::Block*
 return st;
 }
 if (!_block_queue.empty()) {
-auto fblock = _block_queue.front();
-block->swap(*fblock.get());
+const BlockData block_data = _block_queue.front();
+block->swap(*block_data.block);
 *find_block = true

Re: [PR] [cherry-pick](branch-2.1) Pick "Use async group commit rpc call (#36499)" [doris]

2024-07-07 Thread via GitHub



dataroaring merged PR #37380:
URL: https://github.com/apache/doris/pull/37380


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris) branch branch-2.1 updated: [cherry-pick](branch-2.1) Pick "Use async group commit rpc call (#36499)" (#37380)

2024-07-07 Thread dataroaring

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
 new 89857d3780b [cherry-pick](branch-2.1) Pick "Use async group commit rpc 
call (#36499)" (#37380)
89857d3780b is described below

commit 89857d3780b9a4c952dbac2532458a5fd8edddc6
Author: abmdocrt 
AuthorDate: Sun Jul 7 18:28:19 2024 +0800

[cherry-pick](branch-2.1) Pick "Use async group commit rpc call (#36499)" 
(#37380)

## Proposed changes

Issue Number: close #xxx



Pick #36499
---
 be/src/service/internal_service.cpp | 19 +--
 1 file changed, 5 insertions(+), 14 deletions(-)

diff --git a/be/src/service/internal_service.cpp 
b/be/src/service/internal_service.cpp
index 466eaa03cb5..c6bedc630e8 100644
--- a/be/src/service/internal_service.cpp
+++ b/be/src/service/internal_service.cpp
@@ -,23 +,19 @@ void 
PInternalServiceImpl::group_commit_insert(google::protobuf::RpcController*
 ctx->pipe = pipe;
 Status st = _exec_env->new_load_stream_mgr()->put(load_id, ctx);
 if (st.ok()) {
-std::mutex mutex;
-std::condition_variable cv;
-bool handled = false;
 try {
 st = _exec_plan_fragment_impl(
 request->exec_plan_fragment_request().request(),
 request->exec_plan_fragment_request().version(),
 request->exec_plan_fragment_request().compact(),
-[&](RuntimeState* state, Status* status) {
+[&, response, done, load_id](RuntimeState* state, 
Status* status) {
+brpc::ClosureGuard cb_closure_guard(done);
 response->set_label(state->import_label());
 response->set_txn_id(state->wal_id());
 
response->set_loaded_rows(state->num_rows_load_success());
 
response->set_filtered_rows(state->num_rows_load_filtered());
-st = *status;
-std::unique_lock l(mutex);
-handled = true;
-cv.notify_one();
+status->to_protobuf(response->mutable_status());
+_exec_env->new_load_stream_mgr()->remove(load_id);
 });
 } catch (const Exception& e) {
 st = e.to_status();
@@ -2249,6 +2245,7 @@ void 
PInternalServiceImpl::group_commit_insert(google::protobuf::RpcController*
 if (!st.ok()) {
 LOG(WARNING) << "exec plan fragment failed, errmsg=" << st;
 } else {
+closure_guard.release();
 for (int i = 0; i < request->data().size(); ++i) {
 std::unique_ptr row(new PDataRow());
 row->CopyFrom(request->data(i));
@@ -2259,15 +2256,9 @@ void 
PInternalServiceImpl::group_commit_insert(google::protobuf::RpcController*
 }
 if (st.ok()) {
 static_cast(pipe->finish());
-std::unique_lock l(mutex);
-if (!handled) {
-cv.wait(l);
-}
 }
 }
 }
-st.to_protobuf(response->mutable_status());
-_exec_env->new_load_stream_mgr()->remove(load_id);
 });
 if (!ret) {
 _exec_env->new_load_stream_mgr()->remove(load_id);


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [branch-2.1](routine-load) optimize out of range error message [doris]

2024-07-07 Thread via GitHub



dataroaring merged PR #37391:
URL: https://github.com/apache/doris/pull/37391


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris) branch branch-2.1 updated: [branch-2.1](routine-load) optimize out of range error message (#37391)

2024-07-07 Thread dataroaring

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
 new 423483ed8ff [branch-2.1](routine-load) optimize out of range error 
message (#37391)
423483ed8ff is described below

commit 423483ed8ff165e3c1e9a9f8d9a051a41de575d4
Author: hui lai <1353307...@qq.com>
AuthorDate: Sun Jul 7 18:29:04 2024 +0800

[branch-2.1](routine-load) optimize out of range error message (#37391)

## Proposed changes
pick #36450

before
```
ErrorReason{code=errCode = 105, msg='be 10002 abort task, task id: 
d846f3d3-7c9e-44a7-bee0-3eff8cd11c6f job id: 11310 with reason: 
[INTERNAL_ERROR]Offset out of range,

0#  doris::Status doris::Status::Error<6, 
true>(std::basic_string_view >) at 
/mnt/disk1/laihui/doris/be/src/common/status.h:422
1#  doris::Status 
doris::Status::InternalError(std::basic_string_view >) at /mnt/disk1/laihui/doris/be/src/common/status.h:468
2#  
doris::KafkaDataConsumer::group_consume(doris::BlockingQueue*,
 long) at 
/mnt/disk1/laihui/doris/be/src/runtime/routine_load/data_consumer.cpp:226
3#  
doris::KafkaDataConsumerGroup::actual_consume(std::shared_ptr,
 doris::BlockingQueue*, long, std::function) at 
/mnt/disk1/laihui/doris/be/src/runtime/routine_load/data_consumer_group.cpp:200
4#  void std::__invoke_impl, 
doris::BlockingQueue*, long, std::function), doris::KafkaDataConsumerGroup*&, 
std::shared_ptr&, 
doris::BlockingQueue*&, long&, 
doris::KafkaDataConsumerGroup::start_all(std::shared_ptr,
 std::shared_ptr)::$_0&>(std::__invoke_m [...]
...
```

now
```
ErrorReason{code=errCode = 105, msg='be 10002 abort task, task id: 
3ba0c0f4-d13c-4dfa-90ce-3df922fd9340 job id: 11310 with reason: 
[INTERNAL_ERROR]Offset out of range, consume partition 0, consume offset 100, 
the offset used by job does not exist in kafka, please check the offset, using 
the Alter ROUTINE LOAD command to modify it, and resume the job'}
```

## Proposed changes

Issue Number: close #xxx


---
 be/src/runtime/routine_load/data_consumer.cpp  |  22 +++-
 .../routine_load/data/test_out_of_range.csv|   1 +
 .../routine_load/test_out_of_range_error.groovy| 112 +
 3 files changed, 134 insertions(+), 1 deletion(-)

diff --git a/be/src/runtime/routine_load/data_consumer.cpp 
b/be/src/runtime/routine_load/data_consumer.cpp
index 9c40e85e281..d185a610af7 100644
--- a/be/src/runtime/routine_load/data_consumer.cpp
+++ b/be/src/runtime/routine_load/data_consumer.cpp
@@ -36,6 +36,7 @@
 #include "runtime/small_file_mgr.h"
 #include "service/backend_options.h"
 #include "util/blocking_queue.hpp"
+#include "util/debug_points.h"
 #include "util/defer_op.h"
 #include "util/stopwatch.hpp"
 #include "util/string_util.h"
@@ -218,6 +219,16 @@ Status 
KafkaDataConsumer::group_consume(BlockingQueue* queue,
 consumer_watch.start();
 std::unique_ptr msg(_k_consumer->consume(1000 /* 
timeout, ms */));
 consumer_watch.stop();
+DBUG_EXECUTE_IF("KafkaDataConsumer.group_consume.out_of_range", {
+done = true;
+std::stringstream ss;
+ss << "Offset out of range"
+   << ", consume partition " << msg->partition() << ", consume 
offset "
+   << msg->offset();
+LOG(WARNING) << "kafka consume failed: " << _id << ", msg: " << 
ss.str();
+st = Status::InternalError(ss.str());
+break;
+});
 switch (msg->err()) {
 case RdKafka::ERR_NO_ERROR:
 if (_consuming_partition_ids.count(msg->partition()) <= 0) {
@@ -258,10 +269,19 @@ Status 
KafkaDataConsumer::group_consume(BlockingQueue* queue,
 }
 break;
 }
+case RdKafka::ERR_OFFSET_OUT_OF_RANGE: {
+done = true;
+std::stringstream ss;
+ss << msg->errstr() << ", consume partition " << msg->partition() 
<< ", consume offset "
+   << msg->offset();
+LOG(WARNING) << "kafka consume failed: " << _id << ", msg: " << 
ss.str();
+st = Status::InternalError(ss.str());
+break;
+}
 default:
 LOG(WARNING) << "kafka consume failed: " << _id << ", msg: " << 
msg->errstr();
 done = true;
-st = Status::InternalError(msg->errstr());
+st = Status::InternalError(msg->errstr());
 break;
 }
 
diff --git 
a/regression-test/suites/load_p0/routine_load/data/test_out_of_range.csv 
b/regression-test/suites/load_p0/routine_load/data/test_out_of_range.csv
new file mode 100644
index 000..e2ad18c3878
--- /dev/null
+++ b/regression-test/suites/load_p0/routine_load/data/test_out_of_range.csv
@@ -0,0 +1 @@
+routine_lo

[PR] [fix](Recycler) Add missing new operator for recycler plugin [doris]

2024-07-07 Thread via GitHub



ByteYue opened a new pull request, #37393:
URL: https://github.com/apache/doris/pull/37393

   ## Proposed changes
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](Recycler) Add missing new operator for recycler plugin [doris]

2024-07-07 Thread via GitHub



ByteYue commented on PR #37393:
URL: https://github.com/apache/doris/pull/37393#issuecomment-2212401135

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](Recycler) Add missing new operator for recycler plugin [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37393:
URL: https://github.com/apache/doris/pull/37393#issuecomment-2212401520

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](Recycler) Add missing new operator for recycler plugin [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37393:
URL: https://github.com/apache/doris/pull/37393#issuecomment-2212401511

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [branch-2.1] Picks "[Fix](delete) Fix delete job timeout when executing delete from ... #37363" [doris]

2024-07-07 Thread via GitHub



dataroaring merged PR #37374:
URL: https://github.com/apache/doris/pull/37374


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris) branch branch-2.1 updated: [branch-2.1] Picks "[Fix](delete) Fix delete job timeout when executing delete from ... #37363" (#37374)

2024-07-07 Thread dataroaring

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
 new a05406ecc9d [branch-2.1] Picks "[Fix](delete) Fix delete job timeout 
when executing delete from ... #37363" (#37374)
a05406ecc9d is described below

commit a05406ecc9dd1362ca9613a4bdc945e63ad05615
Author: bobhan1 
AuthorDate: Sun Jul 7 18:33:17 2024 +0800

[branch-2.1] Picks "[Fix](delete) Fix delete job timeout when executing 
delete from ... #37363" (#37374)

## Proposed changes

picks https://github.com/apache/doris/pull/37363
---
 be/src/olap/delete_handler.cpp |  5 ++
 .../main/java/org/apache/doris/load/DeleteJob.java |  6 +++
 .../java/org/apache/doris/master/MasterImpl.java   | 28 +++---
 .../main/java/org/apache/doris/task/PushTask.java  | 12 +
 .../test_delete_from_timeout.groovy| 59 ++
 5 files changed, 104 insertions(+), 6 deletions(-)

diff --git a/be/src/olap/delete_handler.cpp b/be/src/olap/delete_handler.cpp
index 6e390874126..8d85eb84bab 100644
--- a/be/src/olap/delete_handler.cpp
+++ b/be/src/olap/delete_handler.cpp
@@ -35,6 +35,7 @@
 #include "olap/predicate_creator.h"
 #include "olap/tablet_schema.h"
 #include "olap/utils.h"
+#include "util/debug_points.h"
 
 using apache::thrift::ThriftDebugString;
 using std::vector;
@@ -90,6 +91,10 @@ std::string trans_op(const std::string& opt) {
 Status DeleteHandler::generate_delete_predicate(const TabletSchema& schema,
 const std::vector& 
conditions,
 DeletePredicatePB* del_pred) {
+DBUG_EXECUTE_IF("DeleteHandler::generate_delete_predicate.inject_failure", 
{
+return Status::Error(dp->param("error_code"),
+dp->param("error_msg"));
+})
 if (conditions.empty()) {
 return Status::Error(
 "invalid parameters for store_cond. condition_size={}", 
conditions.size());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/DeleteJob.java 
b/fe/fe-core/src/main/java/org/apache/doris/load/DeleteJob.java
index f9c94284db8..761db0f4725 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/DeleteJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/DeleteJob.java
@@ -373,6 +373,12 @@ public class DeleteJob extends 
AbstractTxnStateChangeCallback implements DeleteJ
 long timeoutMs = getTimeoutMs();
 boolean ok = countDownLatch.await(timeoutMs, TimeUnit.MILLISECONDS);
 if (ok) {
+if (!countDownLatch.getStatus().ok()) {
+// encounter some errors that don't need to retry, abort 
directly
+LOG.warn("delete job failed, errmsg={}", 
countDownLatch.getStatus().getErrorMsg());
+throw new UserException(String.format("delete job failed, 
errmsg:%s",
+countDownLatch.getStatus().getErrorMsg()));
+}
 return;
 }
 
diff --git a/fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java 
b/fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java
index 485463d8daf..12d908ff317 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java
@@ -137,7 +137,8 @@ public class MasterImpl {
 && taskType != TTaskType.DOWNLOAD && taskType != 
TTaskType.MOVE
 && taskType != TTaskType.CLONE && taskType != 
TTaskType.PUBLISH_VERSION
 && taskType != TTaskType.CREATE && taskType != 
TTaskType.UPDATE_TABLET_META_INFO
-&& taskType != TTaskType.STORAGE_MEDIUM_MIGRATE) {
+&& taskType != TTaskType.STORAGE_MEDIUM_MIGRATE
+&& taskType != TTaskType.REALTIME_PUSH) {
 return result;
 }
 }
@@ -150,7 +151,6 @@ public class MasterImpl {
 finishCreateReplica(task, request);
 break;
 case REALTIME_PUSH:
-checkHasTabletInfo(request);
 Preconditions.checkState(request.isSetReportVersion());
 finishRealtimePush(task, request);
 break;
@@ -295,16 +295,32 @@ public class MasterImpl {
 }
 }
 
-private void finishRealtimePush(AgentTask task, TFinishTaskRequest 
request) {
-List finishTabletInfos = request.getFinishTabletInfos();
-Preconditions.checkState(finishTabletInfos != null && 
!finishTabletInfos.isEmpty());
-
+private void finishRealtimePush(AgentTask task, TFinishTaskRequest 
request) throws Exception {
 PushTask pushTask = (PushTask) task;
 
 long dbId = pushTask

Re: [PR] [branch-2.1](routine-load) increase routine load job default max batch size and rows [doris]

2024-07-07 Thread via GitHub



dataroaring merged PR #37388:
URL: https://github.com/apache/doris/pull/37388


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris) branch branch-2.1 updated: [branch-2.1](routine-load) increase routine load job default max batch size and rows (#37388)

2024-07-07 Thread dataroaring

This is an automated email from the ASF dual-hosted git repository.

dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.1 by this push:
 new 97e4025ee05 [branch-2.1](routine-load) increase routine load job 
default max batch size and rows  (#37388)
97e4025ee05 is described below

commit 97e4025ee055a26bc1d4986a21780a714dad409a
Author: hui lai <1353307...@qq.com>
AuthorDate: Sun Jul 7 18:35:08 2024 +0800

[branch-2.1](routine-load) increase routine load job default max batch size 
and rows  (#37388)

pick #36632

Most users only care about the size of **max_batch_interval**, but in
order to achieve an interval effect, they have to configure
**max_batch_rows** and **max_batch_size** according to the
characteristics of the data. By adjusting these two default values,
users do not need to worry about configuration in most scenarios.

## Proposed changes

Issue Number: close #xxx


---
 .../main/java/org/apache/doris/load/routineload/RoutineLoadJob.java   | 4 ++--
 .../java/org/apache/doris/load/routineload/RoutineLoadJobTest.java| 3 +--
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
 
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
index 130bd87b018..0a85bc63ac1 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
@@ -109,8 +109,8 @@ public abstract class RoutineLoadJob extends 
AbstractTxnStateChangeCallback impl
 public static final double DEFAULT_MAX_FILTER_RATIO = 1.0;
 
 public static final long DEFAULT_MAX_INTERVAL_SECOND = 10;
-public static final long DEFAULT_MAX_BATCH_ROWS = 20;
-public static final long DEFAULT_MAX_BATCH_SIZE = 100 * 1024 * 1024; // 
100MB
+public static final long DEFAULT_MAX_BATCH_ROWS = 2000;
+public static final long DEFAULT_MAX_BATCH_SIZE = 1024 * 1024 * 1024; // 
1GB
 public static final long DEFAULT_EXEC_MEM_LIMIT = 2 * 1024 * 1024 * 1024L;
 public static final boolean DEFAULT_STRICT_MODE = false; // default is 
false
 public static final int DEFAULT_SEND_BATCH_PARALLELISM = 1;
diff --git 
a/fe/fe-core/src/test/java/org/apache/doris/load/routineload/RoutineLoadJobTest.java
 
b/fe/fe-core/src/test/java/org/apache/doris/load/routineload/RoutineLoadJobTest.java
index d9494374c03..863cc6807c7 100644
--- 
a/fe/fe-core/src/test/java/org/apache/doris/load/routineload/RoutineLoadJobTest.java
+++ 
b/fe/fe-core/src/test/java/org/apache/doris/load/routineload/RoutineLoadJobTest.java
@@ -355,7 +355,6 @@ public class RoutineLoadJobTest {
 11, "localhost:9092", "test_topic", UserIdentity.ADMIN);
 Deencapsulation.setField(routineLoadJob, "maxErrorNum", 10);
 Deencapsulation.setField(routineLoadJob, "maxBatchRows", 10);
-Deencapsulation.setField(routineLoadJob, "maxBatchRows", 10);
 String showCreateInfo = routineLoadJob.getShowCreateInfo();
 String expect = "CREATE ROUTINE LOAD test_load ON 11\n"
 + "WITH APPEND\n"
@@ -366,7 +365,7 @@ public class RoutineLoadJobTest {
 + "\"max_filter_ratio\" = \"1.0\",\n"
 + "\"max_batch_interval\" = \"10\",\n"
 + "\"max_batch_rows\" = \"10\",\n"
-+ "\"max_batch_size\" = \"104857600\",\n"
++ "\"max_batch_size\" = \"1073741824\",\n"
 + "\"format\" = \"csv\",\n"
 + "\"strip_outer_array\" = \"false\",\n"
 + "\"num_as_string\" = \"false\",\n"


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [feature](function) add function from_simple_format to VecDateTime class and DateV2 [doris]

2024-07-07 Thread via GitHub



zclllyybb commented on PR #32065:
URL: https://github.com/apache/doris/pull/32065#issuecomment-2212409636

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix] fix date hardcode to date function [doris]

2024-07-07 Thread via GitHub



133tosakarin commented on PR #37389:
URL: https://github.com/apache/doris/pull/37389#issuecomment-2212428878

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[I] [Enhancement] (mtmv) Support read sync materialized view in async materialized view [doris]

2024-07-07 Thread via GitHub



liutang123 opened a new issue, #37394:
URL: https://github.com/apache/doris/issues/37394

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no 
similar issues.
   
   
   ### Description
   
   When creating an asynchronous materialized view, if a synchronous 
materialized view is selected, the following error will be reported：
   ```
   ERROR 1105 (HY000): errCode = 2, detailMessage = Unable to find a suitable 
base table for partitioning, the fail reason is can't not find valid partition 
track column, because related base table partition column doesn't contain the 
mv partition or partition nullable check fail, the mvReferenceColumn is `mv_dt` 
DATEV2 NULL
   ```
   
   ### Solution
   
   Identify the column in async materialized view definition. If it selected a 
column from the sync materialized view, find the real column from the olap 
table.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [opt](inverted index) opt value extraction from column to string [doris]

2024-07-07 Thread via GitHub



zzzxl1993 opened a new pull request, #37395:
URL: https://github.com/apache/doris/pull/37395

   ## Proposed changes
   
   Issue Number: close #xxx
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [Enhancement] (mtmv) Support read sync materialized view in async materialized view [doris]

2024-07-07 Thread via GitHub



liutang123 opened a new pull request, #37396:
URL: https://github.com/apache/doris/pull/37396

   Identify the column in async materialized view definition. If it selected a 
column from a sync materialized view, find the real column from the olap table 
and check if it is the partition column of the olap table.
   
   ## Proposed changes
   
   Issue Number: close #37394
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [Enhancement] (mtmv) Support read sync materialized view in async materialized view [doris]

2024-07-07 Thread via GitHub



liutang123 commented on PR #37396:
URL: https://github.com/apache/doris/pull/37396#issuecomment-2212432785

   Will add regression tests after reviewed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [opt](inverted index) opt value extraction from column to string [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37395:
URL: https://github.com/apache/doris/pull/37395#issuecomment-2212433361

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [Enhancement](mtmv) Support read sync materialized view in async materialized view [doris]

2024-07-07 Thread via GitHub



liutang123 commented on PR #37396:
URL: https://github.com/apache/doris/pull/37396#issuecomment-2212433987

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [fix](nereids) Pick the stop watch is not reset (#37168) [doris]

2024-07-07 Thread via GitHub



mymeiyi opened a new pull request, #37397:
URL: https://github.com/apache/doris/pull/37397

   When use JDBC to insert into Doris, after execute several sqls, will get 
`Nereids cost too much time (> 30s )`, but the audit log shows that the sql 
only cost several milliseconds. The reason is that the stop watch is never 
reset.
   
   ## Proposed changes
   
   Issue Number: close #xxx
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](nereids) Pick the stop watch is not reset (#37168) [doris]

2024-07-07 Thread via GitHub



mymeiyi commented on PR #37397:
URL: https://github.com/apache/doris/pull/37397#issuecomment-2212440251

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](case) fix struct format out files [doris]

2024-07-07 Thread via GitHub



morningman commented on PR #37350:
URL: https://github.com/apache/doris/pull/37350#issuecomment-2212441728

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [improve](group commit) Pick Modify group commit case and modify cancel status (#35995) [doris]

2024-07-07 Thread via GitHub



mymeiyi opened a new pull request, #37398:
URL: https://github.com/apache/doris/pull/37398

   We get many same logs when group commit is cancelled: ``` 
group_commit_mgr.cpp:182] cancel group commit, instance_id=xx, label=xx, 
status=[OK]
   ```
   the problem is that `runtime_state->is_cancelled` is true, but 
`runtime_state->cancel_reason()` is `OK`
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [improve](group commit) Pick Modify group commit case and modify cancel status (#35995) [doris]

2024-07-07 Thread via GitHub



mymeiyi commented on PR #37398:
URL: https://github.com/apache/doris/pull/37398#issuecomment-2212445145

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [improve](group commit) Pick Modify group commit case and modify cancel status (#35995) [doris]

2024-07-07 Thread via GitHub



dataroaring commented on PR #37398:
URL: https://github.com/apache/doris/pull/37398#issuecomment-2212445206

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [improve](group commit) Pick Modify group commit case and modify cancel status (#35995) [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37398:
URL: https://github.com/apache/doris/pull/37398#issuecomment-2212446911

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [branch-2.1](routine-load) make get Kafka meta timeout configurable [doris]

2024-07-07 Thread via GitHub



sollhui opened a new pull request, #37399:
URL: https://github.com/apache/doris/pull/37399

   pick #36619
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [branch-2.1](routine-load) make get Kafka meta timeout configurable [doris]

2024-07-07 Thread via GitHub



sollhui commented on PR #37399:
URL: https://github.com/apache/doris/pull/37399#issuecomment-2212447371

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [branch-2.1](routine-load) fix auto resume invalid when FE leader change [doris]

2024-07-07 Thread via GitHub



sollhui opened a new pull request, #37400:
URL: https://github.com/apache/doris/pull/37400

   pick https://github.com/apache/doris/pull/37071
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [branch-2.1](routine-load) fix auto resume invalid when FE leader change [doris]

2024-07-07 Thread via GitHub



sollhui commented on PR #37400:
URL: https://github.com/apache/doris/pull/37400#issuecomment-2212448815

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [branch-2.1](routine-load) fix auto resume invalid when FE leader change [doris]

2024-07-07 Thread via GitHub



sollhui commented on PR #37400:
URL: https://github.com/apache/doris/pull/37400#issuecomment-2212448881

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [branch-2.1](routine-load) make get Kafka meta timeout configurable [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37399:
URL: https://github.com/apache/doris/pull/37399#issuecomment-2212449190

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [branch-2.1](load) fix no error url if no partition can be found (#36831) [doris]

2024-07-07 Thread via GitHub



sollhui opened a new pull request, #37401:
URL: https://github.com/apache/doris/pull/37401

   ## Proposed changes
   
   pick #36831
   
   before
   ```
   Stream load result: {
   "TxnId": 2014,
   "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
   "Comment": "",
   "TwoPhaseCommit": "false",
   "Status": "Fail",
   "Message": "[DATA_QUALITY_ERROR]Encountered unqualified data, stop 
processing",
   "NumberTotalRows": 1,
   "NumberLoadedRows": 1,
   "NumberFilteredRows": 0,
   "NumberUnselectedRows": 0,
   "LoadBytes": 1669,
   "LoadTimeMs": 58,
   "BeginTxnTimeMs": 0,
   "StreamLoadPutTimeMs": 10,
   "ReadDataTimeMs": 0,
   "WriteDataTimeMs": 47,
   "CommitAndPublishTimeMs": 0
   }
   ```
   
   after
   ```
   Stream load result: {
   "TxnId": 2014,
   "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
   "Comment": "",
   "TwoPhaseCommit": "false",
   "Status": "Fail",
   "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
   "NumberTotalRows": 1,
   "NumberLoadedRows": 0,
   "NumberFilteredRows": 1,
   "NumberUnselectedRows": 0,
   "LoadBytes": 1669,
   "LoadTimeMs": 58,
   "BeginTxnTimeMs": 0,
   "StreamLoadPutTimeMs": 10,
   "ReadDataTimeMs": 0,
   "WriteDataTimeMs": 47,
   "CommitAndPublishTimeMs": 0,
   "ErrorURL": 
"http://:8040/api/_load_error_log?file=__shard_4/error_log_insert_stmt_c6461270125a615b-2873833fb48d56a3_c6461270125a615b_2873833fb48d56a3";
   }
   ```
   
   ## Proposed changes
   
   Issue Number: close #xxx
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [branch-2.1](load) fix no error url if no partition can be found (#36831) [doris]

2024-07-07 Thread via GitHub



sollhui commented on PR #37401:
URL: https://github.com/apache/doris/pull/37401#issuecomment-2212450346

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [branch-2.1](query) print query id when killed by timeout checker [doris]

2024-07-07 Thread via GitHub



sollhui opened a new pull request, #37402:
URL: https://github.com/apache/doris/pull/37402

   ## Proposed changes
   
   pick #36868
   
   ```
   2024-06-26 14:58:30,917 WARN (connect-scheduler-check-timer-0|92) 
[ConnectContext.checkTimeout():776] kill query timeout, remote: :XX, query 
timeout: 90
   ```
   It is hard to know which query is killed when timeout, so it is necessary to 
print query id.
   
   ## Proposed changes
   
   Issue Number: close #xxx
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [branch-2.1](query) print query id when killed by timeout checker [doris]

2024-07-07 Thread via GitHub



sollhui commented on PR #37402:
URL: https://github.com/apache/doris/pull/37402#issuecomment-2212450830

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](nereids) Pick the stop watch is not reset (#37168) [doris]

2024-07-07 Thread via GitHub



mymeiyi commented on PR #37397:
URL: https://github.com/apache/doris/pull/37397#issuecomment-2212450833

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [Fix] Error: missing field initializer [doris]

2024-07-07 Thread via GitHub



wyxxxcat opened a new pull request, #37403:
URL: https://github.com/apache/doris/pull/37403

   ## Proposed changes
   
   Issue Number: close #xxx
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [Fix] Error: missing field initializer [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37403:
URL: https://github.com/apache/doris/pull/37403#issuecomment-2212452194

    `sh-checker report`
   
   To get the full details, please check in the 
[job]("https://github.com/apache/doris/actions/runs/9827706324";) output.
   
   
   shellcheck errors
   
   ```
   
   'shellcheck ' returned error 1 finding the following syntactical issues:
   
   --
   
   In build.sh line 289:
   local submodule_path=$1
   ^-^ SC2317 (info): Command appears to be 
unreachable. Check usage (or ignore if invoked indirectly).
   
   
   In build.sh line 290:
   local submodule_name=$2
   ^-^ SC2317 (info): Command appears to be 
unreachable. Check usage (or ignore if invoked indirectly).
   
   
   In build.sh line 291:
   local archive_url=$3
   ^--^ SC2317 (info): Command appears to be unreachable. 
Check usage (or ignore if invoked indirectly).
   
   
   In build.sh line 293:
   set +e
   ^^ SC2317 (info): Command appears to be unreachable. Check usage (or 
ignore if invoked indirectly).
   
   
   In build.sh line 294:
   cd "${DORIS_HOME}"
   ^^ SC2317 (info): Command appears to be unreachable. 
Check usage (or ignore if invoked indirectly).
   
   
   In build.sh line 295:
   echo "Update ${submodule_name} submodule ..."
   ^-- SC2317 (info): Command appears to be unreachable. Check usage (or 
ignore if invoked indirectly).
   
   
   In build.sh line 296:
   git submodule update --init --recursive "${submodule_path}"
   ^-- SC2317 (info): Command appears to be unreachable. Check usage (or 
ignore if invoked indirectly).
   
   
   In build.sh line 297:
   exit_code=$?
   ^--^ SC2317 (info): Command appears to be unreachable. Check 
usage (or ignore if invoked indirectly).
   
   
   In build.sh line 298:
   if [[ "${exit_code}" -eq 0 ]]; then
   ^-- SC2317 (info): Command appears to be unreachable. Check usage (or 
ignore if invoked indirectly).
  ^^ SC2317 (info): Command appears to be 
unreachable. Check usage (or ignore if invoked indirectly).
   
   
   In build.sh line 299:
   cd "${submodule_path}"
   ^^ SC2317 (info): Command appears to be 
unreachable. Check usage (or ignore if invoked indirectly).
   
   
   In build.sh line 300:
   submodule_commit_id=$(git rev-parse HEAD)
   ^-- SC2317 (info): Command appears to be unreachable. Check usage 
(or ignore if invoked indirectly).
 ^^ SC2317 (info): Command 
appears to be unreachable. Check usage (or ignore if invoked indirectly).
   
   
   In build.sh line 301:
   cd -
   ^--^ SC2317 (info): Command appears to be unreachable. Check usage 
(or ignore if invoked indirectly).
   
   
   In build.sh line 302:
   expect_submodule_commit_id=$(git ls-tree HEAD "${submodule_path}" | 
awk '{print $3}')
   ^-- SC2317 (info): Command appears to be unreachable. Check usage 
(or ignore if invoked indirectly).
^-- SC2317 (info): Command appears to 
be unreachable. Check usage (or ignore if invoked indirectly).
   
   
   In build.sh line 303:
   echo "Current commit ID of ${submodule_name} submodule: 
${submodule_commit_id}, expected is ${expect_submodule_commit_id}"
   ^-- SC2317 (info): Command appears to be unreachable. Check usage 
(or ignore if invoked indirectly).
   
   
   In build.sh line 305:
   set -e
   ^^ SC2317 (info): Command appears to be unreachable. Check usage (or 
ignore if invoked indirectly).
   
   
   In build.sh line 306:
   if [[ "${exit_code}" -ne 0 ]]; then
   ^-- SC2317 (info): Command appears to be unreachable. Check usage (or 
ignore if invoked indirectly).
  ^^ SC2317 (info): Command appears to be 
unreachable. Check usage (or ignore if invoked indirectly).
   
   
   In build.sh line 307:
   set +e
   ^^ SC2317 (info): Command appears to be unreachable. Check usage 
(or ignore if invoked indirectly).
   
   
   In build.sh line 309:
   submodule_commit=$(git ls-tree HEAD "${submodule_path}" | awk 
'{print $3}')
   ^-- SC2317 (info): Command appears to be unreachable. Check usage 
(or ignore if invoked indirectly).
  ^-- SC2317 (info): Command appears to be 
unreachable. Check usage (or ignore if invoked indirectly).
   
   
   In build.sh line 310:
   exit_code=$?
   ^--^ SC2317 (info): Command appears to be unreachable. Check 
usage (or ignore if invoked indirectly).
   
   
   In build.sh line 311:
   if [[ "${exit_code}" = "0" ]]; then
   ^-- SC2317 (info): Command appears to be unreachable. Check usage 
(or ignore if invoked indirectly).
  ^

Re: [PR] [branch-2.1](load) fix no error url if no partition can be found (#36831) [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37401:
URL: https://github.com/apache/doris/pull/37401#issuecomment-2212452275

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [fix](group commit) Pick make group commit cancel in time (#36249) [doris]

2024-07-07 Thread via GitHub



mymeiyi opened a new pull request, #37404:
URL: https://github.com/apache/doris/pull/37404

   pick https://github.com/apache/doris/pull/36249/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](group commit) Pick make group commit cancel in time (#36249) [doris]

2024-07-07 Thread via GitHub



mymeiyi commented on PR #37404:
URL: https://github.com/apache/doris/pull/37404#issuecomment-2212453527

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [Fix] Error: missing field initializer [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37403:
URL: https://github.com/apache/doris/pull/37403#issuecomment-2212454386

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [Fix] Error: missing field initializer [doris]

2024-07-07 Thread via GitHub



wyxxxcat commented on PR #37403:
URL: https://github.com/apache/doris/pull/37403#issuecomment-2212454848

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](group commit) Pick make group commit cancel in time (#36249) [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37404:
URL: https://github.com/apache/doris/pull/37404#issuecomment-2212455076

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [Fix] Error: missing field initializer [doris]

2024-07-07 Thread via GitHub



wyxxxcat commented on PR #37403:
URL: https://github.com/apache/doris/pull/37403#issuecomment-2212455963

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [fix](group commit) Pick Fix the incorrect group commit count in log; fix the core in get_first_block (#36408) [doris]

2024-07-07 Thread via GitHub



mymeiyi opened a new pull request, #37405:
URL: https://github.com/apache/doris/pull/37405

   Pick https://github.com/apache/doris/pull/36408/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](group commit) Pick Fix the incorrect group commit count in log; fix the core in get_first_block (#36408) [doris]

2024-07-07 Thread via GitHub



mymeiyi commented on PR #37405:
URL: https://github.com/apache/doris/pull/37405#issuecomment-2212456138

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

[PR] [performance](load) do not copy input_block in memtable (#36939) [doris]

2024-07-07 Thread via GitHub



kaijchen opened a new pull request, #37407:
URL: https://github.com/apache/doris/pull/37407

   cherry-pick #36939


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [performance](load) do not copy input_block in memtable (#36939) [doris]

2024-07-07 Thread via GitHub



kaijchen commented on PR #37407:
URL: https://github.com/apache/doris/pull/37407#issuecomment-2212456988

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [performance](move-memtable) only call _select_streams when necessary (#35576) [doris]

2024-07-07 Thread via GitHub



kaijchen commented on PR #37406:
URL: https://github.com/apache/doris/pull/37406#issuecomment-2212457091

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [Fix] Error: missing field initializer [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37403:
URL: https://github.com/apache/doris/pull/37403#issuecomment-2212457878

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](group commit) Pick Fix the incorrect group commit count in log; fix the core in get_first_block (#36408) [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37405:
URL: https://github.com/apache/doris/pull/37405#issuecomment-2212457971

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [performance](move-memtable) only call _select_streams when necessary (#35576) [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37406:
URL: https://github.com/apache/doris/pull/37406#issuecomment-2212458512

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [performance](load) do not copy input_block in memtable (#36939) [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37407:
URL: https://github.com/apache/doris/pull/37407#issuecomment-2212458691

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [I] [Bug] array_range函数官方文档生成日期数组报错 [doris]

2024-07-07 Thread via GitHub



superdiaodiao commented on issue #37329:
URL: https://github.com/apache/doris/issues/37329#issuecomment-2212470332

   Which branch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [enhancement](regression-test) dup modify key [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37390:
URL: https://github.com/apache/doris/pull/37390#issuecomment-2212470656

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [enhancement](regression-test) dup modify key [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #37390:
URL: https://github.com/apache/doris/pull/37390#issuecomment-2212470662

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](Recycler) Add missing new operator for recycler plugin [doris]

2024-07-07 Thread via GitHub



ByteYue commented on PR #37393:
URL: https://github.com/apache/doris/pull/37393#issuecomment-2212476418

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](move-memtable) check segment num when closing each tablet [doris]

2024-07-07 Thread via GitHub



kaijchen commented on PR #36753:
URL: https://github.com/apache/doris/pull/36753#issuecomment-2212509101

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

Re: [PR] [fix](move-memtable) check segment num when closing each tablet [doris]

2024-07-07 Thread via GitHub



github-actions[bot] commented on PR #36753:
URL: https://github.com/apache/doris/pull/36753#issuecomment-2212511105

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

(doris-website) branch asf-site updated (510ffc80813 -> 2f54d89bb7c)

2024-07-07 Thread github-bot

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/doris-website.git


 discard 510ffc80813 Automated deployment with doris branch @ 
eaa52d3f4cb750f8751e448be6929e711e2ed079
 new 2f54d89bb7c Automated deployment with doris branch @ 
eaa52d3f4cb750f8751e448be6929e711e2ed079

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (510ffc80813)
\
 N -- N -- N   refs/heads/asf-site (2f54d89bb7c)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 docs/1.2/search-index.json   | 2 +-
 docs/2.0/search-index.json   | 2 +-
 docs/dev/search-index.json   | 2 +-
 search-index.json| 2 +-
 zh-CN/docs/1.2/search-index.json | 2 +-
 zh-CN/docs/2.0/search-index.json | 2 +-
 zh-CN/docs/dev/search-index.json | 2 +-
 zh-CN/search-index.json  | 2 +-
 8 files changed, 8 insertions(+), 8 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org

1 2 3 4 5 >

1 - 100 of 476 matches

Mail list logo