[GitHub] [doris] ChPi commented on pull request #12959: [Bug](bitmap) intersect_count function use in string cause ASAN error (#11936)
ChPi commented on PR #12959: URL: https://github.com/apache/doris/pull/12959#issuecomment-1257576085 是不是合错了，1.1.2已经release，该是1.1-lts？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new issue, #12964: [Enhancement] SelectRollupTest UT is unstable
jackwener opened a new issue, #12964: URL: https://github.com/apache/doris/issues/12964 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description ``` org.apache.doris.common.DdlException: errCode = 2, detailMessage = Table[t]'s state is not NORMAL. Do not allow doing ALTER ops at org.apache.doris.nereids.rules.mv.SelectRollupTest.runBeforeAll(SelectRollupTest.java:52) ``` ### Solution _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luzhijing commented on issue #11706: Good First Issue
luzhijing commented on issue #11706: URL: https://github.com/apache/doris/issues/11706#issuecomment-1257586346 > @luzhijing I have Translated and made the PR. Can you please review it? Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #12965: Buffer pool dcheck
dataroaring opened a new pull request, #12965: URL: https://github.com/apache/doris/pull/12965 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yixiutt opened a new issue, #12966: [Enhancement] support vertical compaction
yixiutt opened a new issue, #12966: URL: https://github.com/apache/doris/issues/12966 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description Compaction in some case such as wide table will cost a lot memory, every row will be loaded in compaction, so memory cost will be rowset_num * segment_num * column_num * page_size when every column load first page, default page size is 64K, so if a table has 1000 columns it will cost rowset_num * segment_num * 64M。 Vertical compaction can handle compaction in column groups so not cost to much memory. I'll accomplish this work in few weeks. ### Solution Vertical Compaction, steps to do: 1. framework, support duplicate key vertical compaction, including all basic functions and can run correctly. 2. unique key 3. agg key 4. some optimize such as RowSourceBuffer write to tmp file to support large compaction. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz merged pull request #12965: Buffer pool dcheck
xinyiZzz merged PR #12965: URL: https://github.com/apache/doris/pull/12965 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: Buffer pool dcheck (#12965)
This is an automated email from the ASF dual-hosted git repository.
zouxinyi pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new b4a86fc0b2 Buffer pool dcheck (#12965)
b4a86fc0b2 is described below
commit b4a86fc0b2fe9c522dec67337d2bfe1da88c5878
Author: Yongqiang YANG <98214048+dataroar...@users.noreply.github.com>
AuthorDate: Mon Sep 26 15:31:03 2022 +0800
Buffer pool dcheck (#12965)
fix core and deadlick
---
be/src/runtime/bufferpool/buffer_allocator.cc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/runtime/bufferpool/buffer_allocator.cc
b/be/src/runtime/bufferpool/buffer_allocator.cc
index ed1d156df5..38869a9da9 100644
--- a/be/src/runtime/bufferpool/buffer_allocator.cc
+++ b/be/src/runtime/bufferpool/buffer_allocator.cc
@@ -365,7 +365,7 @@ int64_t BufferPool::BufferAllocator::ScavengeBuffers(bool
slow_but_sure, int cur
if (slow_but_sure && bytes_found < target_bytes) {
bytes_found +=
DecreaseBytesRemaining(target_bytes - bytes_found, true,
&system_bytes_remaining_);
-DCHECK_EQ(bytes_found, target_bytes) << DebugString();
+// DCHECK_EQ(bytes_found, target_bytes) << DebugString();
}
return bytes_found;
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz closed pull request #12962: [fix](memory) Fix buffer pool deadlock
xinyiZzz closed pull request #12962: [fix](memory) Fix buffer pool deadlock URL: https://github.com/apache/doris/pull/12962 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zxealous opened a new pull request, #12967: [typo](config) Fix typos in comments
zxealous opened a new pull request, #12967: URL: https://github.com/apache/doris/pull/12967 # Proposed changes Issue Number: close #xxx ## Problem summary Fix typos in comments ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on pull request #12818: [refactor](fe-core src test catalog): refactor and replace use NIO
jackwener commented on PR #12818: URL: https://github.com/apache/doris/pull/12818#issuecomment-1257667311 Thanks @dinggege1024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #12970: [Improvement](outfile) Support output null in parquet writer
Gabriel39 opened a new pull request, #12970: URL: https://github.com/apache/doris/pull/12970 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jacktengg opened a new pull request, #12973: [improvement](storage) For debugging problems: add session variable skip_storage_engine_merge to treat agg and unique data model as dup mo
jacktengg opened a new pull request, #12973: URL: https://github.com/apache/doris/pull/12973 Mannual pick from master (#11952) For debug purpose: Add session variable skip_storage_engine_merge, when set to true, tables of aggregate key model and unique key model will be read as duplicate key model. Add session variable skip_delete_predicate, when set to true, rows deleted with delete statement will be selected. # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12818: [refactor](fe-core src test catalog): refactor and replace use NIO
github-actions[bot] commented on PR #12818: URL: https://github.com/apache/doris/pull/12818#issuecomment-1257689003 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #12974: [branch-1.1-lts](cherry-pick) Remove mem tracker record mem pool actual memory usage
xinyiZzz opened a new pull request, #12974: URL: https://github.com/apache/doris/pull/12974 # Proposed changes Issue Number: close #xxx ## Problem summary cherry-pick #12954 ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener merged pull request #12818: [refactor](fe-core src test catalog): refactor and replace use NIO
jackwener merged PR #12818: URL: https://github.com/apache/doris/pull/12818 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [refactor](fe-core src test catalog): refactor and replace use NIO #12818 (#12818)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3902b2bfad [refactor](fe-core src test catalog): refactor and replace
use NIO #12818 (#12818)
3902b2bfad is described below
commit 3902b2bfad8be2675c573bd480654b3fe50aefb3
Author: DingGeGe <109070189+dinggege1...@users.noreply.github.com>
AuthorDate: Mon Sep 26 16:51:46 2022 +0800
[refactor](fe-core src test catalog): refactor and replace use NIO #12818
(#12818)
---
.../org/apache/doris/catalog/AdminStmtTest.java| 37 +++---
.../java/org/apache/doris/catalog/BackendTest.java | 16 +-
.../doris/catalog/ColocateTableIndexTest.java | 16 +-
.../doris/catalog/ColumnGsonSerializationTest.java | 30 ++
.../org/apache/doris/catalog/ColumnStatsTest.java | 15 -
.../org/apache/doris/catalog/ColumnTypeTest.java | 15 -
.../org/apache/doris/catalog/DiskInfoTest.java | 17 +-
.../doris/catalog/MaterializedIndexMetaTest.java | 21 ++--
.../doris/catalog/MaterializedIndexTest.java | 15 -
.../doris/catalog/OdbcCatalogResourceTest.java | 15 -
.../org/apache/doris/catalog/OlapTableTest.java| 1 +
.../org/apache/doris/catalog/PartitionKeyTest.java | 15 -
.../doris/catalog/ReplicaAllocationTest.java | 15 -
.../java/org/apache/doris/catalog/ReplicaTest.java | 15 -
.../org/apache/doris/catalog/ResourceMgrTest.java | 1 -
.../org/apache/doris/catalog/S3ResourceTest.java | 15 -
16 files changed, 127 insertions(+), 132 deletions(-)
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/catalog/AdminStmtTest.java
b/fe/fe-core/src/test/java/org/apache/doris/catalog/AdminStmtTest.java
index 3be8952841..c43e08f08f 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/AdminStmtTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/AdminStmtTest.java
@@ -26,15 +26,15 @@ import
org.apache.doris.persist.SetReplicaStatusOperationLog;
import org.apache.doris.utframe.TestWithFeService;
import com.google.common.collect.Lists;
-import org.junit.Assert;
+import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import java.io.DataInputStream;
import java.io.DataOutputStream;
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
import java.util.List;
public class AdminStmtTest extends TestWithFeService {
@@ -55,9 +55,9 @@ public class AdminStmtTest extends TestWithFeService {
@Test
public void testAdminSetReplicaStatus() throws Exception {
Database db =
Env.getCurrentInternalCatalog().getDbNullable("default_cluster:test");
-Assert.assertNotNull(db);
+Assertions.assertNotNull(db);
OlapTable tbl = (OlapTable) db.getTableNullable("tbl1");
-Assert.assertNotNull(tbl);
+Assertions.assertNotNull(tbl);
// tablet id, backend id
List> tabletToBackendList = Lists.newArrayList();
for (Partition partition : tbl.getPartitions()) {
@@ -69,11 +69,11 @@ public class AdminStmtTest extends TestWithFeService {
}
}
}
-Assert.assertEquals(3, tabletToBackendList.size());
+Assertions.assertEquals(3, tabletToBackendList.size());
long tabletId = tabletToBackendList.get(0).first;
long backendId = tabletToBackendList.get(0).second;
Replica replica = Env.getCurrentInvertedIndex().getReplica(tabletId,
backendId);
-Assert.assertFalse(replica.isBad());
+Assertions.assertFalse(replica.isBad());
// set replica to bad
String adminStmt = "admin set replica status properties ('tablet_id' =
'" + tabletId + "', 'backend_id' = '"
@@ -81,7 +81,7 @@ public class AdminStmtTest extends TestWithFeService {
AdminSetReplicaStatusStmt stmt = (AdminSetReplicaStatusStmt)
parseAndAnalyzeStmt(adminStmt);
Env.getCurrentEnv().setReplicaStatus(stmt);
replica = Env.getCurrentInvertedIndex().getReplica(tabletId,
backendId);
-Assert.assertTrue(replica.isBad());
+Assertions.assertTrue(replica.isBad());
// set replica to ok
adminStmt = "admin set replica status properties ('tablet_id' = '" +
tabletId + "', 'backend_id' = '"
@@ -89,17 +89,17 @@ public class AdminStmtTest extends TestWithFeService {
stmt = (AdminSetReplicaStatusStmt) parseAndAnalyzeStmt(adminStmt);
Env.getCurrentEnv().setReplicaStatus(stmt);
replica = Env.getCurrentInvertedIndex().getReplica(tabletId,
backendId);
-Assert.assertFalse(replica.isBad());
+Assertions.assertFalse(replica.isBad());
[GitHub] [doris] github-actions[bot] commented on pull request #12151: [Nereids][Feature] Constant fold of an expression
github-actions[bot] commented on PR #12151: URL: https://github.com/apache/doris/pull/12151#issuecomment-1257708359 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12151: [Nereids][Feature] Constant fold of an expression
github-actions[bot] commented on PR #12151: URL: https://github.com/apache/doris/pull/12151#issuecomment-1257708409 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen opened a new pull request, #12975: [enhancement](load) avoid duplicate reduce on same TabletsChannel
zhannngchen opened a new pull request, #12975: URL: https://github.com/apache/doris/pull/12975 # Proposed changes Issue Number: close #xxx ## Problem summary When reaching the hard limit, there might be multiple threads can pick same LoadChannel and call reduce_mem_usage on same TabletsChannel. Although there's a lock and condition variable can prevent multiple threads to reduce mem usage concurrently, but they still can do same reduce-work on that channel multiple times one by one, even it's just reduced. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 1018ji commented on issue #11875: [Bug] sql execute error after change the replication_num in colocate table
1018ji commented on issue #11875: URL: https://github.com/apache/doris/issues/11875#issuecomment-1257723627 大佬这问题咋解决的啊 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #12974: [branch-1.1-lts](cherry-pick) Remove mem tracker record mem pool actual memory usage
dataroaring merged PR #12974: URL: https://github.com/apache/doris/pull/12974 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [branch-1.1-lts](cherry-pick) Remove mem tracker record mem pool actual memory usage (#12974)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new efe29795d5 [branch-1.1-lts](cherry-pick) Remove mem tracker record mem
pool actual memory usage (#12974)
efe29795d5 is described below
commit efe29795d53968e5bc219e5f647279a95a4dc72e
Author: Xinyi Zou
AuthorDate: Mon Sep 26 17:14:05 2022 +0800
[branch-1.1-lts](cherry-pick) Remove mem tracker record mem pool actual
memory usage (#12974)
---
be/src/runtime/mem_pool.cpp | 15 +--
be/src/runtime/mem_pool.h | 14 --
be/src/runtime/memory/mem_tracker.cpp | 7 ++-
be/src/runtime/memory/mem_tracker_limiter.cpp | 8 ++--
be/src/runtime/memory/mem_tracker_limiter.h | 17 +++--
5 files changed, 16 insertions(+), 45 deletions(-)
diff --git a/be/src/runtime/mem_pool.cpp b/be/src/runtime/mem_pool.cpp
index 21b475400e..050614eb5f 100644
--- a/be/src/runtime/mem_pool.cpp
+++ b/be/src/runtime/mem_pool.cpp
@@ -41,13 +41,11 @@ MemPool::ChunkInfo::ChunkInfo(const Chunk& chunk_) :
chunk(chunk_), allocated_by
DorisMetrics::instance()->memory_pool_bytes_total->increment(chunk.size);
}
-
MemPool::MemPool(const std::string& label)
: current_chunk_idx_(-1),
next_chunk_size_(INITIAL_CHUNK_SIZE),
total_allocated_bytes_(0),
- total_reserved_bytes_(0),
- peak_allocated_bytes_(0) {
+ total_reserved_bytes_(0) {
mem_tracker_own_ = MemTracker::CreateTracker(-1, label + ":MemPool");
mem_tracker_ = mem_tracker_own_.get();
}
@@ -59,9 +57,6 @@ MemPool::~MemPool() {
ChunkAllocator::instance()->free(chunk.chunk);
}
mem_tracker_->Release(total_bytes_released);
-THREAD_MEM_TRACKER_TRANSFER_FROM(total_bytes_released -
peak_allocated_bytes_,
-
ExecEnv::GetInstance()->orphan_mem_tracker_raw());
-
DorisMetrics::instance()->memory_pool_bytes_total->increment(-total_bytes_released);
}
@@ -81,15 +76,11 @@ void MemPool::free_all() {
total_bytes_released += chunk.chunk.size;
ChunkAllocator::instance()->free(chunk.chunk);
}
-THREAD_MEM_TRACKER_TRANSFER_FROM(total_bytes_released -
peak_allocated_bytes_,
-
ExecEnv::GetInstance()->orphan_mem_tracker_raw());
-
chunks_.clear();
next_chunk_size_ = INITIAL_CHUNK_SIZE;
current_chunk_idx_ = -1;
total_allocated_bytes_ = 0;
total_reserved_bytes_ = 0;
-peak_allocated_bytes_ = 0;
mem_tracker_->Release(total_bytes_released);
DorisMetrics::instance()->memory_pool_bytes_total->increment(-total_bytes_released);
@@ -148,7 +139,6 @@ bool MemPool::find_chunk(size_t min_size, bool
check_limits) {
mem_tracker_->Release(chunk_size);
return false;
}
-THREAD_MEM_TRACKER_TRANSFER_TO(chunk_size,
ExecEnv::GetInstance()->orphan_mem_tracker_raw());
ASAN_POISON_MEMORY_REGION(chunk.data, chunk_size);
// Put it before the first free chunk. If no free chunks, it goes at the
end.
if (first_free_idx == static_cast(chunks_.size())) {
@@ -214,8 +204,6 @@ void MemPool::acquire_data(MemPool* src, bool keep_current)
{
src->total_allocated_bytes_ = 0;
}
-reset_peak();
-
if (!keep_current) src->free_all();
DCHECK(src->check_integrity(false));
DCHECK(check_integrity(false));
@@ -228,7 +216,6 @@ void MemPool::exchange_data(MemPool* other) {
std::swap(next_chunk_size_, other->next_chunk_size_);
std::swap(total_allocated_bytes_, other->total_allocated_bytes_);
std::swap(total_reserved_bytes_, other->total_reserved_bytes_);
-std::swap(peak_allocated_bytes_, other->peak_allocated_bytes_);
std::swap(chunks_, other->chunks_);
// update MemTracker
diff --git a/be/src/runtime/mem_pool.h b/be/src/runtime/mem_pool.h
index 8eee3334ce..11853ebfc5 100644
--- a/be/src/runtime/mem_pool.h
+++ b/be/src/runtime/mem_pool.h
@@ -96,7 +96,6 @@ public:
next_chunk_size_(INITIAL_CHUNK_SIZE),
total_allocated_bytes_(0),
total_reserved_bytes_(0),
- peak_allocated_bytes_(0),
mem_tracker_(mem_tracker) {
DCHECK(mem_tracker != nullptr);
}
@@ -161,7 +160,6 @@ public:
int64_t total_allocated_bytes() const { return total_allocated_bytes_; }
int64_t total_reserved_bytes() const { return total_reserved_bytes_; }
-int64_t peak_allocated_bytes() const { return peak_allocated_bytes_; }
MemTracker* mem_tracker() { return mem_tracker_; }
@@ -207,14 +205,6 @@ private:
/// data. Otherwise the current chunk can be either empty or full.
bool check_integrity(bool check_current_chunk_empty);
-void reset_peak() {
-if (total_allocated_by
[GitHub] [doris] morrySnow merged pull request #12151: [Nereids][Feature] Constant fold of an expression
morrySnow merged PR #12151: URL: https://github.com/apache/doris/pull/12151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #12976: [DOC](datev2) Add documents for DateV2
Gabriel39 opened a new pull request, #12976: URL: https://github.com/apache/doris/pull/12976 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] lsy3993 commented on pull request #12957: [fix](planner) : Incorrect result of CAST AS CHAR
lsy3993 commented on PR #12957: URL: https://github.com/apache/doris/pull/12957#issuecomment-1257751483 This bug fix is only for the case of casting constant string in FE. When the cast statement calculates in BE, we may fix it later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg opened a new pull request, #12977: [chore](regression-test) add default group(p0) for regression-test
mrhhsg opened a new pull request, #12977: URL: https://github.com/apache/doris/pull/12977 # Proposed changes Issue Number: close #xxx ## Problem Summary It's difficult to run all groups of regression-test on local env, so make `run-regression-test.sh` will just run p0 by default. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #12961: [fix](column)fix get_shrinked_column misspell
morningman merged PR #12961: URL: https://github.com/apache/doris/pull/12961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (7977bebfed -> 35076431ab)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 7977bebfed [feature](Nereids) constant expression folding (#12151) add 35076431ab [fix](column)fix get_shrinked_column misspell (#12961) No new revisions were added by this update. Summary of changes: be/src/vec/columns/column.h| 2 +- be/src/vec/columns/column_array.cpp| 4 ++-- be/src/vec/columns/column_array.h | 2 +- be/src/vec/columns/column_jsonb.cpp| 2 +- be/src/vec/columns/column_jsonb.h | 2 +- be/src/vec/columns/column_nullable.cpp | 4 ++-- be/src/vec/columns/column_nullable.h | 2 +- be/src/vec/columns/column_string.cpp | 2 +- be/src/vec/columns/column_string.h | 2 +- be/src/vec/core/block.cpp | 2 +- 10 files changed, 12 insertions(+), 12 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LemonLiTree opened a new pull request, #12978: [typo](docs)Add bitmap_count doc And Adjustment function list
LemonLiTree opened a new pull request, #12978: URL: https://github.com/apache/doris/pull/12978 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jacktengg opened a new pull request, #12979: [regression-case](improve) improve regression test case
jacktengg opened a new pull request, #12979: URL: https://github.com/apache/doris/pull/12979 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee opened a new pull request, #12980: [Bug](bitmap) intersect_count function use in string cause ASAN error (#11936)
HappenLee opened a new pull request, #12980: URL: https://github.com/apache/doris/pull/12980 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #12981: [improvement](memory) set TCMALLOC_HEAP_LIMIT_MB to control memory co…
dataroaring opened a new pull request, #12981: URL: https://github.com/apache/doris/pull/12981 …nsumption of tcmalloc # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12967: [typo](config) Fix typos in comments
github-actions[bot] commented on PR #12967: URL: https://github.com/apache/doris/pull/12967#issuecomment-1257802316 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12967: [typo](config) Fix typos in comments
github-actions[bot] commented on PR #12967: URL: https://github.com/apache/doris/pull/12967#issuecomment-1257802373 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dinggege1024 opened a new pull request, #12982: [fix](fe-core) sparkload comment mislead which file types it support
dinggege1024 opened a new pull request, #12982: URL: https://github.com/apache/doris/pull/12982 # Proposed changes Issue Number: open #12895 ## Problem summary Del the misleading comment about spark load file formats. Actually, spark load supports orc\csv\parq file formats. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [✅] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [✅] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [✅] No Need 4. Does it need to update dependencies: - [ ] Yes - [✅] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [✅] No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dinggege1024 commented on pull request #12982: [fix](fe-core) sparkload comment mislead which file types it support
dinggege1024 commented on PR #12982: URL: https://github.com/apache/doris/pull/12982#issuecomment-1257821421 @jackwener Hi jack,can you help review the pr? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12975: [enhancement](load) avoid duplicate reduce on same TabletsChannel
github-actions[bot] commented on PR #12975: URL: https://github.com/apache/doris/pull/12975#issuecomment-1257832540 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12975: [enhancement](load) avoid duplicate reduce on same TabletsChannel
github-actions[bot] commented on PR #12975: URL: https://github.com/apache/doris/pull/12975#issuecomment-1257832578 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 closed pull request #12545: [fix](doc)Fix sql function doc
Gabriel39 closed pull request #12545: [fix](doc)Fix sql function doc URL: https://github.com/apache/doris/pull/12545 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zxealous closed issue #12598: [Enhancement](cache) Add remote segment cache statistics to profile
zxealous closed issue #12598: [Enhancement](cache) Add remote segment cache statistics to profile URL: https://github.com/apache/doris/issues/12598 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zxealous closed pull request #12597: (enhancement)(cache) Add remote segment cache statistics to profile
zxealous closed pull request #12597: (enhancement)(cache) Add remote segment cache statistics to profile URL: https://github.com/apache/doris/pull/12597 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luozenglin opened a new pull request, #12983: [fix](planner) fix push down no grouping agg
luozenglin opened a new pull request, #12983: URL: https://github.com/apache/doris/pull/12983 The value column of the agg does not support zone_map index, fixing the value column pushing down to zone map causes null pointer. # Proposed changes Issue Number: close #12958 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] AshinGau opened a new pull request, #12984: [feature-wip](parquet-reader) bugfix, VFileScanner has memory leakage when ParquetReader init failed
AshinGau opened a new pull request, #12984: URL: https://github.com/apache/doris/pull/12984 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener merged pull request #12967: [typo](config) Fix typos in comments
jackwener merged PR #12967: URL: https://github.com/apache/doris/pull/12967 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: fix doc typos (#12967)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 595a5337dc fix doc typos (#12967) 595a5337dc is described below commit 595a5337dcbe52ccde4d18222c4f90ae60e0b0f0 Author: zxealous AuthorDate: Mon Sep 26 20:11:26 2022 +0800 fix doc typos (#12967) --- be/src/common/config.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/be/src/common/config.h b/be/src/common/config.h index 5b5eccdf0d..4bc79cf9b0 100644 --- a/be/src/common/config.h +++ b/be/src/common/config.h @@ -732,7 +732,7 @@ CONF_mInt32(max_segment_num_per_rowset, "200"); // The connection timeout when connecting to external table such as odbc table. CONF_mInt32(external_table_connect_timeout_sec, "30"); -// The capacity of lur cache in segment loader. +// The capacity of lru cache in segment loader. // Althought it is called "segment cache", but it caches segments in rowset granularity. // So the value of this config should corresponding to the number of rowsets on this BE. CONF_mInt32(segment_cache_capacity, "100"); - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 closed pull request #12749: [Improvement](storage) Use columnwise merge sort to accelerate MergeIterator
Gabriel39 closed pull request #12749: [Improvement](storage) Use columnwise merge sort to accelerate MergeIterator URL: https://github.com/apache/doris/pull/12749 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #12807: [Enhancement](optimize) optimize for insert_indices_from
HappenLee commented on code in PR #12807:
URL: https://github.com/apache/doris/pull/12807#discussion_r979950567
##
be/src/vec/columns/column_vector.cpp:
##
@@ -349,20 +349,22 @@ void ColumnVector::insert_range_from(const IColumn&
src, size_t start, size_t
template
void ColumnVector::insert_indices_from(const IColumn& src, const int*
indices_begin,
const int* indices_end) {
-const Self& src_vec = assert_cast(src);
auto origin_size = size();
auto new_size = indices_end - indices_begin;
data.resize(origin_size + new_size);
-for (int i = 0; i < new_size; ++i) {
-int offset = indices_begin[i];
-if constexpr (std::is_same_v) {
-// Now Uint8 use to identify null and non null
-// 1. nullable column : offset == -1 means is null at the here,
set true here
-// 2. real data column : offset == -1 what at is meaningless
-data[origin_size + i] = (offset == -1) ? T {1} :
src_vec.get_element(offset);
-} else {
-data[origin_size + i] = (offset == -1) ? T {0} :
src_vec.get_element(offset);
+const T* src_data = reinterpret_cast(src.get_raw_data().data);
+
+if constexpr (std::is_same_v) {
+// nullmap : indices_begin[i] == -1 means is null at the here, set
true here
+for (int i = 0; i < new_size; ++i) {
+data[origin_size + i] = (indices_begin[i] == -1) +
+(indices_begin[i] != -1) *
src_data[indices_begin[i]];
+}
+} else {
+// real data : indices_begin[i] == -1 what at is meaningless
+for (int i = 0; i < new_size; ++i) {
+data[origin_size + i] = src_data[indices_begin[i]];
Review Comment:
This is undefined behavior, and might break horribly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #8864: [feature-wip](statistics) step6: statistics is available
Kikyou1997 commented on code in PR #8864:
URL: https://github.com/apache/doris/pull/8864#discussion_r979964422
##
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStats.java:
##
@@ -72,16 +75,94 @@ public class ColumnStats {
private float avgSize = -1; // in bytes
private long maxSize = -1; // in bytes
private long numNulls = -1;
-private LiteralExpr minValue;
-private LiteralExpr maxValue;
-
-public static ColumnStats createDefaultColumnStats() {
-ColumnStats columnStats = new ColumnStats();
-columnStats.setAvgSize(1);
-columnStats.setMaxSize(1);
-columnStats.setNdv(1);
-columnStats.setNumNulls(0);
-return columnStats;
+
+private LiteralExpr minValue = new NullLiteral();
+private LiteralExpr maxValue = new NullLiteral();
+
+/**
+ * Return default column statistic.
+ */
+public static ColumnStats getDefaultColumnStats() {
+return new ColumnStats();
+}
+
+/**
+ * Merge column statistics(the original statistics should not be modified)
+ *
+ * @param left statistics to be merged
+ * @param right statistics to be merged
+ */
+public static ColumnStats aggColumnStats(ColumnStats left, ColumnStats
right) {
Review Comment:
rename this method to `mergeColumnStats`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #12985: [feature](Nereids): Eliminate outer join
jackwener opened a new pull request, #12985: URL: https://github.com/apache/doris/pull/12985 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [x] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [x] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ClownXC commented on issue #11706: Good First Issue
ClownXC commented on issue #11706: URL: https://github.com/apache/doris/issues/11706#issuecomment-1257958691 [WeOpen Star] I would like to help TESTS, Please assign me. I want to contribute. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly opened a new pull request, #12986: New star schema
englefly opened a new pull request, #12986: URL: https://github.com/apache/doris/pull/12986 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #8864: [feature-wip](statistics) step6: statistics is available
Kikyou1997 commented on code in PR #8864:
URL: https://github.com/apache/doris/pull/8864#discussion_r979969582
##
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStats.java:
##
@@ -72,16 +75,94 @@ public class ColumnStats {
private float avgSize = -1; // in bytes
private long maxSize = -1; // in bytes
private long numNulls = -1;
-private LiteralExpr minValue;
-private LiteralExpr maxValue;
-
-public static ColumnStats createDefaultColumnStats() {
-ColumnStats columnStats = new ColumnStats();
-columnStats.setAvgSize(1);
-columnStats.setMaxSize(1);
-columnStats.setNdv(1);
-columnStats.setNumNulls(0);
-return columnStats;
+
+private LiteralExpr minValue = new NullLiteral();
+private LiteralExpr maxValue = new NullLiteral();
+
+/**
+ * Return default column statistic.
+ */
+public static ColumnStats getDefaultColumnStats() {
+return new ColumnStats();
+}
+
+/**
+ * Merge column statistics(the original statistics should not be modified)
+ *
+ * @param left statistics to be merged
+ * @param right statistics to be merged
+ */
+public static ColumnStats aggColumnStats(ColumnStats left, ColumnStats
right) {
+// merge ndv
+long leftNdv = left.getNdv();
+long rightNdv = right.getNdv();
+
+if (leftNdv == -1) {
+leftNdv = rightNdv;
+} else {
+leftNdv = rightNdv != -1 ? (leftNdv + rightNdv) : leftNdv;
+}
+
+// merge avg_size
+float leftAvgSize = left.getAvgSize();
+float rightAvgSize = right.getAvgSize();
+if (leftAvgSize == -1) {
+leftAvgSize = rightAvgSize;
+} else {
+leftAvgSize = rightAvgSize != -1 ? ((leftAvgSize + rightAvgSize) /
2) : leftAvgSize;
+}
+
+// merge max_size
+long leftMaxSize = left.getMaxSize();
+long rightMaxSize = right.getMaxSize();
+if (leftMaxSize == -1) {
+leftMaxSize = rightMaxSize;
+} else {
+leftMaxSize = rightMaxSize != -1 ? (Math.max(leftMaxSize,
rightMaxSize)) : leftMaxSize;
Review Comment:
`rightMaxSize != -1` here is redundant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #8864: [feature-wip](statistics) step6: statistics is available
Kikyou1997 commented on code in PR #8864:
URL: https://github.com/apache/doris/pull/8864#discussion_r979972528
##
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStats.java:
##
@@ -72,16 +75,94 @@ public class ColumnStats {
private float avgSize = -1; // in bytes
private long maxSize = -1; // in bytes
private long numNulls = -1;
-private LiteralExpr minValue;
-private LiteralExpr maxValue;
-
-public static ColumnStats createDefaultColumnStats() {
-ColumnStats columnStats = new ColumnStats();
-columnStats.setAvgSize(1);
-columnStats.setMaxSize(1);
-columnStats.setNdv(1);
-columnStats.setNumNulls(0);
-return columnStats;
+
+private LiteralExpr minValue = new NullLiteral();
+private LiteralExpr maxValue = new NullLiteral();
+
+/**
+ * Return default column statistic.
+ */
+public static ColumnStats getDefaultColumnStats() {
+return new ColumnStats();
+}
+
+/**
+ * Merge column statistics(the original statistics should not be modified)
+ *
+ * @param left statistics to be merged
+ * @param right statistics to be merged
+ */
+public static ColumnStats aggColumnStats(ColumnStats left, ColumnStats
right) {
+// merge ndv
+long leftNdv = left.getNdv();
+long rightNdv = right.getNdv();
+
+if (leftNdv == -1) {
+leftNdv = rightNdv;
+} else {
+leftNdv = rightNdv != -1 ? (leftNdv + rightNdv) : leftNdv;
+}
+
+// merge avg_size
+float leftAvgSize = left.getAvgSize();
+float rightAvgSize = right.getAvgSize();
+if (leftAvgSize == -1) {
+leftAvgSize = rightAvgSize;
+} else {
+leftAvgSize = rightAvgSize != -1 ? ((leftAvgSize + rightAvgSize) /
2) : leftAvgSize;
+}
+
+// merge max_size
+long leftMaxSize = left.getMaxSize();
+long rightMaxSize = right.getMaxSize();
+if (leftMaxSize == -1) {
+leftMaxSize = rightMaxSize;
+} else {
+leftMaxSize = rightMaxSize != -1 ? (Math.max(leftMaxSize,
rightMaxSize)) : leftMaxSize;
+}
+
+// merge num_nulls
+long leftNumNulls = left.getNumNulls();
+long rightNumNulls = right.getNumNulls();
+if (leftNumNulls == -1) {
+leftNumNulls = rightNumNulls;
+} else {
+leftNumNulls = rightNumNulls != -1 ? (leftNumNulls +
rightNumNulls) : leftNumNulls;
+}
+
+// merge min_value
+LiteralExpr leftMinValue = left.getMinValue();
+LiteralExpr rightMinValue = right.getMinValue();
+if (leftMinValue == null) {
+leftMinValue = rightMinValue;
+} else {
+leftMinValue = leftMinValue.compareTo(rightMinValue) > 0 ?
rightMinValue : leftMinValue;
Review Comment:
`Math.min` might be more clear
##
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStats.java:
##
@@ -72,16 +75,94 @@ public class ColumnStats {
private float avgSize = -1; // in bytes
private long maxSize = -1; // in bytes
private long numNulls = -1;
-private LiteralExpr minValue;
-private LiteralExpr maxValue;
-
-public static ColumnStats createDefaultColumnStats() {
-ColumnStats columnStats = new ColumnStats();
-columnStats.setAvgSize(1);
-columnStats.setMaxSize(1);
-columnStats.setNdv(1);
-columnStats.setNumNulls(0);
-return columnStats;
+
+private LiteralExpr minValue = new NullLiteral();
+private LiteralExpr maxValue = new NullLiteral();
+
+/**
+ * Return default column statistic.
+ */
+public static ColumnStats getDefaultColumnStats() {
+return new ColumnStats();
+}
+
+/**
+ * Merge column statistics(the original statistics should not be modified)
+ *
+ * @param left statistics to be merged
+ * @param right statistics to be merged
+ */
+public static ColumnStats aggColumnStats(ColumnStats left, ColumnStats
right) {
+// merge ndv
+long leftNdv = left.getNdv();
+long rightNdv = right.getNdv();
+
+if (leftNdv == -1) {
+leftNdv = rightNdv;
+} else {
+leftNdv = rightNdv != -1 ? (leftNdv + rightNdv) : leftNdv;
+}
+
+// merge avg_size
+float leftAvgSize = left.getAvgSize();
+float rightAvgSize = right.getAvgSize();
+if (leftAvgSize == -1) {
+leftAvgSize = rightAvgSize;
+} else {
+leftAvgSize = rightAvgSize != -1 ? ((leftAvgSize + rightAvgSize) /
2) : leftAvgSize;
+}
+
+// merge max_size
+long leftMaxSize = left.getMaxSize();
+long rightMaxSize = right.getMaxSize();
+if (leftMaxSize == -1) {
+leftMaxSize = rightMaxSize;
+} else {
+leftMaxSize = rightMaxSize != -1 ? (Math.m
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #8864: [feature-wip](statistics) step6: statistics is available
Kikyou1997 commented on code in PR #8864:
URL: https://github.com/apache/doris/pull/8864#discussion_r979976545
##
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStats.java:
##
@@ -180,23 +262,10 @@ public void updateStats(Type columnType, Map statsTypeToValue
}
}
-public List getShowInfo() {
-List result = Lists.newArrayList();
-result.add(Long.toString(ndv));
-result.add(Float.toString(avgSize));
-result.add(Long.toString(maxSize));
-result.add(Long.toString(numNulls));
-if (minValue != null) {
-result.add(minValue.getStringValue());
-} else {
-result.add("N/A");
-}
-if (maxValue != null) {
-result.add(maxValue.getStringValue());
-} else {
-result.add("N/A");
-}
-return result;
+// TODO: The generated statistics should not be modified
+public void updateBySelectivity(double selectivity) {
+ndv = (long) Math.ceil(ndv * selectivity);
Review Comment:
Why not use `Math.round`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #8864: [feature-wip](statistics) step6: statistics is available
Kikyou1997 commented on code in PR #8864:
URL: https://github.com/apache/doris/pull/8864#discussion_r979983061
##
fe/fe-core/src/main/java/org/apache/doris/statistics/Statistics.java:
##
@@ -225,12 +199,40 @@ public void mockTableStatsWithRowCount(long tableId, long
rowCount) {
}
if (tableStats.getRowCount() != rowCount) {
-tableStats.setRowCount(rowCount);
+// tableStats.setRowCount(rowCount);
}
}
-// Used for unit test
-public void putTableStats(long id, TableStats tableStats) {
-this.idToTableStats.put(id, tableStats);
+/**
+ * if the table stats is not exist, create a new one.
+ *
+ * @param tableId table id
+ * @return @TableStats
+ */
+private TableStats getNotNullTableStats(long tableId) {
Review Comment:
Do we have machanism to remove the stats of deleted table currently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #8864: [feature-wip](statistics) step6: statistics is available
Kikyou1997 commented on code in PR #8864:
URL: https://github.com/apache/doris/pull/8864#discussion_r979992528
##
fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsJobManager.java:
##
@@ -80,16 +80,24 @@ public Map getIdToStatisticsJob() {
}
public void createStatisticsJob(AnalyzeStmt analyzeStmt) throws
UserException {
-// step1: init statistics job by analyzeStmt
-StatisticsJob statisticsJob =
StatisticsJob.fromAnalyzeStmt(analyzeStmt);
-writeLock();
-try {
-// step2: check restrict
-checkRestrict(analyzeStmt.getDbId(), statisticsJob.getTblIds());
-// step3: create it
-createStatisticsJob(statisticsJob);
-} finally {
-writeUnlock();
+// The current statistics are only used for CBO test,
+// and are not available to users. (work in progress)
+// TODO(wzt): Further tests are needed
+if (Config.enable_cbo_statistics) {
Review Comment:
Set it as a session variables rather than an options in configuration file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 opened a new pull request, #12987: F/stats v2
Kikyou1997 opened a new pull request, #12987: URL: https://github.com/apache/doris/pull/12987 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #8864: [feature-wip](statistics) step6: statistics is available
Kikyou1997 commented on code in PR #8864:
URL: https://github.com/apache/doris/pull/8864#discussion_r980019243
##
fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsJobManager.java:
##
@@ -80,16 +80,24 @@ public Map getIdToStatisticsJob() {
}
public void createStatisticsJob(AnalyzeStmt analyzeStmt) throws
UserException {
-// step1: init statistics job by analyzeStmt
-StatisticsJob statisticsJob =
StatisticsJob.fromAnalyzeStmt(analyzeStmt);
-writeLock();
Review Comment:
Seems like the read lock is never used, so why not use object lock directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Xiaoccer opened a new pull request, #12988: [fix](transfer_thread) fix the loss of notification.
Xiaoccer opened a new pull request, #12988: URL: https://github.com/apache/doris/pull/12988 # Proposed changes ## Problem summary In the VOlapScanNode:: transfer_thread (), the __transfer_done_ should be locked for modification before calling the cv.notify_all(). Otherwise, the thread of VOlapScanNode::get_next() may not know the __transfer_done_ has been modified, and lose the notification before calling cv.wait_for(), which will cause the thread to wait for 1s. We can find that OlapScanNode has locked the __transfer_done_ in the same way. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [X] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [X] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [X] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [X] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [X] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yongjinhou opened a new issue, #12989: [Bug] The value of md5sum(NULL) is not NULL when using broker to import data(Non-vectorized import)
yongjinhou opened a new issue, #12989:
URL: https://github.com/apache/doris/issues/12989
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Version
47797ad7e8a238bbf1d6a07395eac0df51078056
### What's Wrong?
```
CREATE DATABASE test_sys_load_func_strict_test_md5sum_db;
CREATE TABLE
test_sys_load_func_strict_test_md5sum_db.test_sys_load_func_strict_test_md5sum_tb_s
( k1 BIGINT NULL, k2 BIGINT NULL, k3 BIGINT NULL, k4 BIGINT NULL, k5 CHAR(64)
NULL ) DUPLICATE KEY(k1, k2, k3) PARTITION BY RANGE(k1) ( PARTITION partition_a
VALUES LESS THAN ("10"), PARTITION partition_b VALUES LESS THAN
("10"), PARTITION partition_c VALUES LESS THAN ("100"),
PARTITION partition_d VALUES LESS THAN MAXVALUE ) DISTRIBUTED BY HASH(k1)
BUCKETS 13;
LOAD LABEL
test_sys_load_func_strict_test_md5sum_db.label_13_03_51_28_096733_745838664 (
DATA INFILE("hdfs://IP:Port/xxx/timestamp_load_file") INTO TABLE
`test_sys_load_func_strict_test_md5sum_tb_s` PARTITION (partition_a,
partition_b, partition_c, partition_d) COLUMNS TERMINATED BY "," (`k1`, `k2`,
`k3`, `k4`) SET(k5 = md5sum(k1))) WITH BROKER "xxx" ("username"="xxx",
"password"="xxx") PROPERTIES( "max_filter_ratio"="0.05", "strict_mode"="True" );
SELECT * FROM
test_sys_load_func_strict_test_md5sum_db.test_sys_load_func_strict_test_md5sum_tb_s;
```
When above SQL is executed to import data through broker, if k1 is NULL, we
expect k5 is NULL, otherwise, k5 is "d41d8cd98f00b204e9800998ecf8427e".
When select md5sum(NULL) is executed, it does not enter the MD5sum function,
instead, it returns NULL based on the parameter value NULL. When you run
select md5sum (NULL, '111') or select md5sum ('111',NULL), it also does not
enter the MD5sum function, If NULL is contained in the parameter list,NULL is
directly returned.
When data is imported through the broker, a NULL value is passed inside the
md5sum function, resulting in an undesired value
### What You Expected?
When above SQL is executed to import data through broker, if k1 is NULL, we
expect k5 is NULL.
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yongjinhou opened a new pull request, #12990: [fix](Broker Load)fix bug when load data using broker with md5sum()
yongjinhou opened a new pull request, #12990: URL: https://github.com/apache/doris/pull/12990 # Proposed changes Issue Number: close #12989 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [x] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12978: [typo](docs)Add bitmap_count doc And Adjustment function list
github-actions[bot] commented on PR #12978: URL: https://github.com/apache/doris/pull/12978#issuecomment-1258047340 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yongjinhou opened a new issue, #12991: [Bug] The value of sm3sum(NULL) is not NULL when using broker to import data(Non-vectorized import)
yongjinhou opened a new issue, #12991: URL: https://github.com/apache/doris/issues/12991 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 47797ad7e8a238bbf1d6a07395eac0df51078056 ### What's Wrong? The problem is similar to issue https://github.com/apache/doris/issues/12989. ### What You Expected? The problem is similar to issue https://github.com/apache/doris/issues/12989. ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12957: [fix](planner) : Incorrect result of CAST AS CHAR
github-actions[bot] commented on PR #12957: URL: https://github.com/apache/doris/pull/12957#issuecomment-1258066357 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 commented on pull request #12990: [fix](Broker Load)fix bug when load data using broker with md5sum()
Gabriel39 commented on PR #12990: URL: https://github.com/apache/doris/pull/12990#issuecomment-1258066843 @yangzhg I think this is why `md5sum()` doesn't core on row-based engine as described in #12509 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 commented on a diff in pull request #12984: [feature-wip](parquet-reader) bugfix, VFileScanner has memory leakage when ParquetReader init failed
Gabriel39 commented on code in PR #12984:
URL: https://github.com/apache/doris/pull/12984#discussion_r980061621
##
be/src/vec/exec/scan/vfile_scanner.cpp:
##
@@ -336,12 +336,13 @@ Status VFileScanner::_get_next_reader() {
for (int i = 0; i < _file_slot_descs.size(); i++) {
column_names.push_back(_file_slot_descs[i]->col_name());
}
-_cur_reader = new ParquetReader(_profile, _params, range,
column_names,
Review Comment:
I think we should not use `_cur_reader` as a pure pointer. Maybe we can
resolve this issue by changing this to a unique pointer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dinggege1024 commented on issue #12964: [Enhancement] SelectRollupTest UT is unstable
dinggege1024 commented on issue #12964: URL: https://github.com/apache/doris/issues/12964#issuecomment-1258099911 @jackwener Hi jack, i would like to help this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12906: [Bug](function) fix substr return null on row-based engine
github-actions[bot] commented on PR #12906: URL: https://github.com/apache/doris/pull/12906#issuecomment-1258168782 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12906: [Bug](function) fix substr return null on row-based engine
github-actions[bot] commented on PR #12906: URL: https://github.com/apache/doris/pull/12906#issuecomment-1258168740 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] sahilm-10 opened a new pull request, #116: Updation
sahilm-10 opened a new pull request, #116: URL: https://github.com/apache/doris-website/pull/116 Updated xiaomi.md -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Chore](clang) fix some build fail on clang15 (#12882)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8731eea26e [Chore](clang) fix some build fail on clang15 (#12882)
8731eea26e is described below
commit 8731eea26e1b62f8a88ad7ff12b724d4ba629ff5
Author: Pxl
AuthorDate: Mon Sep 26 23:13:28 2022 +0800
[Chore](clang) fix some build fail on clang15 (#12882)
remove unused variables
---
be/CMakeLists.txt | 7 ++-
be/src/agent/task_worker_pool.cpp | 2 --
be/src/exec/partitioned_hash_table.cc | 6 +++---
be/src/exprs/new_agg_fn_evaluator.cc | 2 +-
be/src/exprs/table_function/explode_json_array.cpp | 2 --
be/src/olap/tablet.cpp | 4
be/src/olap/tablet_meta.cpp| 5 -
be/src/util/cgroup_util.cpp| 2 +-
be/src/vec/exec/scan/new_odbc_scanner.cpp | 2 +-
be/src/vec/exec/vodbc_scan_node.cpp| 2 +-
be/src/vec/functions/function.cpp | 3 ---
be/src/vec/functions/nullif.cpp| 2 --
12 files changed, 13 insertions(+), 26 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index db3cc63d4d..115eca0ef5 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -436,7 +436,12 @@ endfunction()
set(CUSTUM_LINKER_COMMAND "ld")
# TODO: mold will link fail on thirdparty brpc now, waiting for investigation.
# TRY_TO_CHANGE_LINKER("mold" "mold")
-TRY_TO_CHANGE_LINKER("lld" "LLD")
+
+# TODO: brpc will link fail on clang-15, need fix in the future.
+if (COMPILER_GCC OR CLANG_VERSION_MAJOR VERSION_LESS 15)
+TRY_TO_CHANGE_LINKER("lld" "LLD")
+endif()
+
TRY_TO_CHANGE_LINKER("gold" "GNU gold")
if (NOT CUSTUM_LINKER_COMMAND STREQUAL "ld")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS}
-fuse-ld=${CUSTUM_LINKER_COMMAND}")
diff --git a/be/src/agent/task_worker_pool.cpp
b/be/src/agent/task_worker_pool.cpp
index bb7e276b2d..91526dab9d 100644
--- a/be/src/agent/task_worker_pool.cpp
+++ b/be/src/agent/task_worker_pool.cpp
@@ -722,14 +722,12 @@ void
TaskWorkerPool::_publish_version_worker_thread_callback() {
.error(status);
finish_task_request.__set_error_tablet_ids(error_tablet_ids);
} else {
-int submit_tablets = 0;
if (config::enable_quick_compaction &&
config::quick_compaction_batch_size > 0) {
for (int i = 0; i < succ_tablet_ids.size(); i++) {
TabletSharedPtr tablet =
StorageEngine::instance()->tablet_manager()->get_tablet(
succ_tablet_ids[i]);
if (tablet != nullptr) {
-submit_tablets++;
tablet->publised_count++;
if (tablet->publised_count %
config::quick_compaction_batch_size == 0) {
StorageEngine::instance()->submit_quick_compaction_task(tablet);
diff --git a/be/src/exec/partitioned_hash_table.cc
b/be/src/exec/partitioned_hash_table.cc
index a662764907..83fe65d1b6 100644
--- a/be/src/exec/partitioned_hash_table.cc
+++ b/be/src/exec/partitioned_hash_table.cc
@@ -439,9 +439,9 @@ void PartitionedHashTable::Close() {
const int64_t HEAVILY_USED = 1024 * 1024;
// TODO: These statistics should go to the runtime profile as well.
if ((num_buckets_ > LARGE_HT) || (num_probes_ > HEAVILY_USED))
VLOG_CRITICAL << PrintStats();
-for (auto& data_page : data_pages_) allocator_->Free(move(data_page));
+for (auto& data_page : data_pages_) allocator_->Free(std::move(data_page));
data_pages_.clear();
-if (bucket_allocation_ != nullptr)
allocator_->Free(move(bucket_allocation_));
+if (bucket_allocation_ != nullptr)
allocator_->Free(std::move(bucket_allocation_));
}
Status PartitionedHashTable::CheckAndResize(uint64_t buckets_to_fill,
@@ -508,7 +508,7 @@ Status PartitionedHashTable::ResizeBuckets(int64_t
num_buckets,
}
num_buckets_ = num_buckets;
-allocator_->Free(move(bucket_allocation_));
+allocator_->Free(std::move(bucket_allocation_));
bucket_allocation_ = std::move(new_allocation);
buckets_ = reinterpret_cast(bucket_allocation_->data());
*got_memory = true;
diff --git a/be/src/exprs/new_agg_fn_evaluator.cc
b/be/src/exprs/new_agg_fn_evaluator.cc
index 1fe0f6e897..3ce6375dde 100644
--- a/be/src/exprs/new_agg_fn_evaluator.cc
+++ b/be/src/exprs/new_agg_fn_evaluator.cc
@@ -183,7 +183,7 @@ Status NewAggFnEvaluator::Open(RuntimeState* state) {
ExprContext* eval = input_evals_[i];
RETURN_IF_ERROR(eval->get_const_value(state, *(agg_fn_.get_child(i)),
&constant_args[i]));
}
-agg_fn_ctx_->impl()->set_constant_args(move(constant_args));
+ag
[GitHub] [doris] BiteTheDDDDt merged pull request #12882: [Chore](clang) support build with clang15
BiteThet merged PR #12882: URL: https://github.com/apache/doris/pull/12882 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] GabbyRichards commented on issue #8721: [Enhancement] Remove some unused include
GabbyRichards commented on issue #8721: URL: https://github.com/apache/doris/issues/8721#issuecomment-1258193378 hi! are there any more header files that need to be removed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jacktengg opened a new pull request, #12992: [function](bitmap) support bitmap_hash64
jacktengg opened a new pull request, #12992: URL: https://github.com/apache/doris/pull/12992 # Proposed changes Issue Number: close #4718 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts created (now 8731eea26e)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-1.2-lts in repository https://gitbox.apache.org/repos/asf/doris.git at 8731eea26e [Chore](clang) fix some build fail on clang15 (#12882) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #12993: [community](asf) add branch-1.2-lts protection
morningman opened a new pull request, #12993: URL: https://github.com/apache/doris/pull/12993 # Proposed changes Issue Number: close #xxx ## Problem summary 1. Add branch-1.2-lts protection Now the branch-1.2-lts is protected same as master branch, all checks should be passed and at least one committer's approval is required. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12807: [Enhancement](optimize) optimize for insert_indices_from
github-actions[bot] commented on PR #12807: URL: https://github.com/apache/doris/pull/12807#issuecomment-1258267655 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12807: [Enhancement](optimize) optimize for insert_indices_from
github-actions[bot] commented on PR #12807: URL: https://github.com/apache/doris/pull/12807#issuecomment-1258267603 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Wilson-BT commented on issue #12718: 【Bug】doris writer write failed on json format
Wilson-BT commented on issue #12718: URL: https://github.com/apache/doris/issues/12718#issuecomment-1258281274 > key.format default value is csv, what mean not match 【DOC】? The code is inconsistent with the official documentation, The old and new versions of the code are not compatible -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #12496: [enhancement](memory) Jemalloc performance optimization and compatibility with MemTracker
xinyiZzz commented on PR #12496: URL: https://github.com/apache/doris/pull/12496#issuecomment-1258322142 > both clickhouse and pingcap has swtiched to jemalloc. [ClickHouse/ClickHouse#2773](https://github.com/ClickHouse/ClickHouse/pull/2773) [pingcap/tiflash#424](https://github.com/pingcap/tiflash/pull/424) I will refer to more later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #12994: [fix](column-const) Avoid return ColumnConst for vliteral
morningman opened a new pull request, #12994: URL: https://github.com/apache/doris/pull/12994 # Proposed changes Issue Number: close #xxx ## Problem summary ColumnConst is hard to use and very error-prone. eg, the `remove_nullable()` method will not remove the `nullable` of inner column data of ColumnConst. And `is_nullable()` method is comment out in ColumnConst with unknown reason. So this PR, I change the default behavior of `vliteral`, to not return ColumnConst. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12983: [fix](planner) fix push down no grouping agg
github-actions[bot] commented on PR #12983: URL: https://github.com/apache/doris/pull/12983#issuecomment-1258359332 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12983: [fix](planner) fix push down no grouping agg
github-actions[bot] commented on PR #12983: URL: https://github.com/apache/doris/pull/12983#issuecomment-1258359388 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12981: [improvement](memory) set TCMALLOC_HEAP_LIMIT_MB to control memory co…
github-actions[bot] commented on PR #12981: URL: https://github.com/apache/doris/pull/12981#issuecomment-1258385518 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12981: [improvement](memory) set TCMALLOC_HEAP_LIMIT_MB to control memory co…
github-actions[bot] commented on PR #12981: URL: https://github.com/apache/doris/pull/12981#issuecomment-1258385565 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12988: [fix](transfer_thread) fix the loss of notification.
github-actions[bot] commented on PR #12988: URL: https://github.com/apache/doris/pull/12988#issuecomment-1258428681 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12988: [fix](transfer_thread) fix the loss of notification.
github-actions[bot] commented on PR #12988: URL: https://github.com/apache/doris/pull/12988#issuecomment-1258428633 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12979: [regression-case](improve) improve regression test case
github-actions[bot] commented on PR #12979: URL: https://github.com/apache/doris/pull/12979#issuecomment-1258447014 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12979: [regression-case](improve) improve regression test case
github-actions[bot] commented on PR #12979: URL: https://github.com/apache/doris/pull/12979#issuecomment-1258447058 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12960: [enhancement](memory) Trigger load channel flush based on process physical memory to avoid OOM
github-actions[bot] commented on PR #12960: URL: https://github.com/apache/doris/pull/12960#issuecomment-1258448647 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12960: [enhancement](memory) Trigger load channel flush based on process physical memory to avoid OOM
github-actions[bot] commented on PR #12960: URL: https://github.com/apache/doris/pull/12960#issuecomment-1258448735 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12993: [community](asf) add branch-1.2-lts protection
github-actions[bot] commented on PR #12993: URL: https://github.com/apache/doris/pull/12993#issuecomment-1258517458 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12993: [community](asf) add branch-1.2-lts protection
github-actions[bot] commented on PR #12993: URL: https://github.com/apache/doris/pull/12993#issuecomment-1258517414 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] closed pull request #8548: fix sort spill memory limit
github-actions[bot] closed pull request #8548: fix sort spill memory limit URL: https://github.com/apache/doris/pull/8548 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12973: [improvement](storage) For debugging problems: add session variable skip_storage_engine_merge to treat agg and unique data model as dup model
yiguolei merged PR #12973: URL: https://github.com/apache/doris/pull/12973 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [improvement](storage) For debugging problems: add session variable skip_storage_engine_merge to treat agg and unique data model as dup model #12973
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new 7b0313d72e [improvement](storage) For debugging problems: add session
variable skip_storage_engine_merge to treat agg and unique data model as dup
model #12973
7b0313d72e is described below
commit 7b0313d72e896c31f294bd507420c9b29c1b18da
Author: TengJianPing <18241664+jackte...@users.noreply.github.com>
AuthorDate: Tue Sep 27 08:32:35 2022 +0800
[improvement](storage) For debugging problems: add session variable
skip_storage_engine_merge to treat agg and unique data model as dup model #12973
For debug purpose:
Add session variable skip_storage_engine_merge, when set to true, tables
of aggregate key model and unique key model will be read as duplicate
key model.
Add session variable skip_delete_predicate, when set to true, rows
deleted with delete statement will be selected.
---
be/src/exec/olap_scanner.cpp | 7 ++-
be/src/olap/reader.cpp | 4 ++--
be/src/runtime/runtime_state.h | 9
docs/en/administrator-guide/variables.md | 8 +++-
docs/zh-CN/administrator-guide/variables.md| 5 +
.../java/org/apache/doris/common/util/Util.java| 4 +++-
.../java/org/apache/doris/qe/SessionVariable.java | 24 ++
gensrc/thrift/PaloInternalService.thrift | 6 ++
8 files changed, 62 insertions(+), 5 deletions(-)
diff --git a/be/src/exec/olap_scanner.cpp b/be/src/exec/olap_scanner.cpp
index a5f29970ea..01f455b46d 100644
--- a/be/src/exec/olap_scanner.cpp
+++ b/be/src/exec/olap_scanner.cpp
@@ -153,7 +153,12 @@ Status OlapScanner::_init_tablet_reader_params(
->rowset_meta()
->is_segments_overlapping());
-_tablet_reader_params.direct_mode = single_version || _aggregation;
+if (_runtime_state->skip_storage_engine_merge()) {
+_tablet_reader_params.direct_mode = true;
+_aggregation = true;
+} else {
+_tablet_reader_params.direct_mode = single_version || _aggregation;
+}
RETURN_IF_ERROR(_init_return_columns(!_tablet_reader_params.direct_mode));
diff --git a/be/src/olap/reader.cpp b/be/src/olap/reader.cpp
index 9b0b7ceec0..c93cbc3971 100644
--- a/be/src/olap/reader.cpp
+++ b/be/src/olap/reader.cpp
@@ -823,8 +823,8 @@ OLAPStatus TabletReader::_init_delete_condition(const
ReaderParams& read_params)
if (read_params.reader_type == READER_CUMULATIVE_COMPACTION) {
return OLAP_SUCCESS;
}
-OLAPStatus ret;
-{
+OLAPStatus ret = OLAP_SUCCESS;
+if (read_params.runtime_state &&
!read_params.runtime_state->skip_delete_predicate()) {
ReadLock rdlock(_tablet->get_header_lock());
ret = _delete_handler.init(_tablet->tablet_schema(),
_tablet->delete_predicates(),
read_params.version.second,
this);
diff --git a/be/src/runtime/runtime_state.h b/be/src/runtime/runtime_state.h
index b9edc56ea0..f96c799898 100644
--- a/be/src/runtime/runtime_state.h
+++ b/be/src/runtime/runtime_state.h
@@ -349,6 +349,15 @@ public:
return _query_options.enable_enable_exchange_node_parallel_merge;
}
+bool skip_storage_engine_merge() const {
+return _query_options.__isset.skip_storage_engine_merge &&
+ _query_options.skip_storage_engine_merge;
+}
+
+bool skip_delete_predicate() const {
+return _query_options.__isset.skip_delete_predicate &&
_query_options.skip_delete_predicate;
+}
+
const std::vector& tablet_commit_infos() const {
return _tablet_commit_infos;
}
diff --git a/docs/en/administrator-guide/variables.md
b/docs/en/administrator-guide/variables.md
index 3cd6b1a718..fb106efa2c 100644
--- a/docs/en/administrator-guide/variables.md
+++ b/docs/en/administrator-guide/variables.md
@@ -496,4 +496,10 @@ Translated with www.DeepL.com/Translator (free version)
SM4_128_CFB128,
SM4_128_OFB,
SM4_128_CTR,
-```
\ No newline at end of file
+```
+
+* `skip_storage_engine_merge`
+ For debugging purpose. In vectorized execution engine, in case of problems
of reading data of Aggregate Key model and Unique Key model, setting value to
`true` will read data as Duplicate Key model.
+
+* `skip_delete_predicate`
+ For debugging purpose. In vectorized execution engine, in case of problems
of reading data, setting value to `true` will also read deleted data.
diff --git a/docs/zh-CN/administrator-guide/variables.md
b/docs/zh-CN/administrator-guide/variables.md
index cef0310456..9c56491b15 100644
--- a/docs/zh-CN/administrator-guide/variables.md
+++ b/docs/zh-CN/administrator-guide/variables.md
@@ -490,3 +490,8 @@ SELECT /*+ SET_VAR(query_timeout = 1,
[GitHub] [doris] yiguolei commented on pull request #12988: [fix](transfer_thread) fix the loss of notification.
yiguolei commented on PR #12988: URL: https://github.com/apache/doris/pull/12988#issuecomment-1258825010 Does new olap scan node also have this problem? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12980: [Bug](bitmap) intersect_count function use in string cause ASAN error (#11936)
yiguolei merged PR #12980: URL: https://github.com/apache/doris/pull/12980 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [Bug](bitmap) intersect_count function use in string cause ASAN error (#11936) (#12980)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new ff48ae51e1 [Bug](bitmap) intersect_count function use in string cause
ASAN error (#11936) (#12980)
ff48ae51e1 is described below
commit ff48ae51e17e16cd0b79b2b5922e40a15c31e259
Author: HappenLee
AuthorDate: Tue Sep 27 08:41:54 2022 +0800
[Bug](bitmap) intersect_count function use in string cause ASAN error
(#11936) (#12980)
---
be/src/util/bitmap_intersect.h | 111 +
.../aggregate_function_orthogonal_bitmap.cpp | 3 +-
.../aggregate_function_orthogonal_bitmap.h | 13 ++-
3 files changed, 122 insertions(+), 5 deletions(-)
diff --git a/be/src/util/bitmap_intersect.h b/be/src/util/bitmap_intersect.h
index dcda6ae5a5..94ff8dc283 100644
--- a/be/src/util/bitmap_intersect.h
+++ b/be/src/util/bitmap_intersect.h
@@ -15,6 +15,8 @@
// specific language governing permissions and limitations
// under the License.
#pragma once
+#include
+
#include "runtime/string_value.h"
#include "udf/udf.h"
#include "util/bitmap_value.h"
@@ -103,6 +105,15 @@ inline char* Helper::write_to(const
StringValue& v, char* dest) {
dest += v.len;
return dest;
}
+
+template <>
+inline char* Helper::write_to(const std::string& v, char* dest) {
+*(uint32_t*)dest = v.size();
+dest += 4;
+memcpy(dest, v.c_str(), v.size());
+dest += v.size();
+return dest;
+}
// write_to end
template <>
@@ -119,6 +130,11 @@ template <>
inline int32_t Helper::serialize_size(const StringValue& v) {
return v.len + 4;
}
+
+template <>
+inline int32_t Helper::serialize_size(const std::string& v) {
+return v.size() + 4;
+}
// serialize_size end
template <>
@@ -147,6 +163,14 @@ inline void Helper::read_from(const char**
src, StringValue* result
*result = StringValue((char*)*src, length);
*src += length;
}
+
+template <>
+inline void Helper::read_from(const char** src, std::string*
result) {
+int32_t length = *(int32_t*)(*src);
+*src += 4;
+*result = std::string((char*)*src, length);
+*src += length;
+}
// read_from end
} // namespace detail
@@ -242,4 +266,91 @@ private:
std::map _bitmaps;
};
+template <>
+struct BitmapIntersect {
+public:
+BitmapIntersect() = default;
+
+explicit BitmapIntersect(const char* src) { deserialize(src); }
+
+void add_key(const std::string_view key) {
+BitmapValue empty_bitmap;
+_bitmaps[key] = empty_bitmap;
+}
+
+void update(const std::string_view& key, const BitmapValue& bitmap) {
+if (_bitmaps.find(key) != _bitmaps.end()) {
+_bitmaps[key] |= bitmap;
+}
+}
+
+void merge(const BitmapIntersect& other) {
+for (auto& kv : other._bitmaps) {
+if (_bitmaps.find(kv.first) != _bitmaps.end()) {
+_bitmaps[kv.first] |= kv.second;
+} else {
+_bitmaps[kv.first] = kv.second;
+}
+}
+}
+
+// intersection
+BitmapValue intersect() const {
+BitmapValue result;
+auto it = _bitmaps.begin();
+result |= it->second;
+it++;
+for (; it != _bitmaps.end(); it++) {
+result &= it->second;
+}
+return result;
+}
+
+// calculate the intersection for _bitmaps's bitmap values
+int64_t intersect_count() const {
+if (_bitmaps.empty()) {
+return 0;
+}
+return intersect().cardinality();
+}
+
+// the serialize size
+size_t size() {
+size_t size = 4;
+for (auto& kv : _bitmaps) {
+size += detail::Helper::serialize_size(kv.first);
+size += kv.second.getSizeInBytes();
+}
+return size;
+}
+
+//must call size() first
+void serialize(char* dest) {
+char* writer = dest;
+*(int32_t*)writer = _bitmaps.size();
+writer += 4;
+for (auto& kv : _bitmaps) {
+writer = detail::Helper::write_to(kv.first, writer);
+kv.second.write(writer);
+writer += kv.second.getSizeInBytes();
+}
+}
+
+void deserialize(const char* src) {
+const char* reader = src;
+int32_t bitmaps_size = *(int32_t*)reader;
+reader += 4;
+for (int32_t i = 0; i < bitmaps_size; i++) {
+std::string key;
+detail::Helper::read_from(&reader, &key);
+BitmapValue bitmap(reader);
+reader += bitmap.getSizeInBytes();
+_bitmaps[key] = bitmap;
+}
+}
+
+private:
+phmap::flat_hash_map _bitmaps;
+};
+
} // namespace doris
diff --git
a/be/src/vec/aggregate_functions/aggregate_function_orthogonal_bitmap.cpp
b/be/src/vec/aggregate_functions/aggregate_function_orthogonal_bitmap.cp
[GitHub] [doris] yiguolei merged pull request #12988: [fix](transfer_thread) fix the loss of notification.
yiguolei merged PR #12988: URL: https://github.com/apache/doris/pull/12988 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
