Re: [PR] [fix](mem) heap-buffer-overflow for function convert_to [doris]
cambyzju commented on PR #46405: URL: https://github.com/apache/doris/pull/46405#issuecomment-2569281224 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](nereids) support median agg function using percentile [doris]
dqz123 commented on PR #46377: URL: https://github.com/apache/doris/pull/46377#issuecomment-2569285297 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) fix file cache potential leakage [doris]
freemandealer commented on PR #46404: URL: https://github.com/apache/doris/pull/46404#issuecomment-2569284855 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap [doris]
hello-stephen commented on PR #46406: URL: https://github.com/apache/doris/pull/46406#issuecomment-2569290669 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap [doris]
englefly opened a new pull request, #46406: URL: https://github.com/apache/doris/pull/46406 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap [doris]
englefly commented on PR #46406: URL: https://github.com/apache/doris/pull/46406#issuecomment-2569291436 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Fix](index build) should not append index to meta while column id is -1 #46307 [doris]
yiguolei merged PR #46335: URL: https://github.com/apache/doris/pull/46335 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap-branch3.0 [doris]
englefly commented on PR #46224: URL: https://github.com/apache/doris/pull/46224#issuecomment-2569297616 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compile) fix fe compile failed by code too large [doris]
hello-stephen commented on PR #46408: URL: https://github.com/apache/doris/pull/46408#issuecomment-2569322238 skip buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compile) fix fe compile failed by code too large [doris]
924060929 commented on PR #46408: URL: https://github.com/apache/doris/pull/46408#issuecomment-2569314658 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compile) fix fe compile failed by code too large [doris]
hello-stephen commented on PR #46408: URL: https://github.com/apache/doris/pull/46408#issuecomment-2569334905 skip buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compile) fix fe compile failed by code too large [doris]
hello-stephen merged PR #46408: URL: https://github.com/apache/doris/pull/46408 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](compile) fix fe compile failed by code too large (#46408)
This is an automated email from the ASF dual-hosted git repository.
hellostephen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8f87a966e57 [fix](compile) fix fe compile failed by code too large
(#46408)
8f87a966e57 is described below
commit 8f87a966e5759b52aa6ed1f848d2ce73519ed4fe
Author: 924060929
AuthorDate: Fri Jan 3 22:42:29 2025 +0800
[fix](compile) fix fe compile failed by code too large (#46408)
### What problem does this PR solve?
fix fe compile failed by code too large
---
.../generator/ExpressionTypeMappingGenerator.java | 23 ++
1 file changed, 23 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/generator/ExpressionTypeMappingGenerator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/generator/ExpressionTypeMappingGenerator.java
index c5a923153df..d84693c00ed 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/generator/ExpressionTypeMappingGenerator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/generator/ExpressionTypeMappingGenerator.java
@@ -36,6 +36,25 @@ import javax.tools.StandardLocation;
/** ExpressionTypeMappingGenerator */
public class ExpressionTypeMappingGenerator {
+private static final Set FORBIDDEN_CLASS = Sets.newHashSet(
+
"org.apache.doris.nereids.trees.expressions.functions.ExpressionTrait",
+"org.apache.doris.nereids.trees.expressions.shape.LeafExpression",
+"org.apache.doris.nereids.trees.expressions.shape.UnaryExpression",
+
"org.apache.doris.nereids.trees.expressions.shape.BinaryExpression",
+
"org.apache.doris.nereids.trees.expressions.functions.AlwaysNullable",
+
"org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable",
+
"org.apache.doris.nereids.trees.expressions.functions.PropagateNullLiteral",
+
"org.apache.doris.nereids.trees.expressions.typecoercion.ImplicitCastInputTypes",
+
"org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature",
+"org.apache.doris.nereids.trees.expressions.functions.Function",
+
"org.apache.doris.nereids.trees.expressions.functions.FunctionTrait",
+
"org.apache.doris.nereids.trees.expressions.functions.ComputeSignature",
+
"org.apache.doris.nereids.trees.expressions.functions.scalar.ScalarFunction",
+
"org.apache.doris.nereids.trees.expressions.typecoercion.ExpectsInputTypes",
+
"org.apache.doris.nereids.trees.expressions.functions.ComputeNullable",
+
"org.apache.doris.nereids.trees.expressions.functions.PropagateNullable"
+);
+
private final JavaAstAnalyzer analyzer;
public ExpressionTypeMappingGenerator(JavaAstAnalyzer javaAstAnalyzer) {
@@ -134,6 +153,10 @@ public class ExpressionTypeMappingGenerator {
+ "= ImmutableMap.builderWithExpectedSize(" +
childrenNameMap.size() + ");\n";
for (String superExpression : superExpressions) {
+if (FORBIDDEN_CLASS.contains(superExpression)) {
+continue;
+}
+
Set childrenClasseSet =
childrenNameMap.get(superExpression)
.stream()
.filter(childClass -> parentNameMap.get(childClass)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression)fix hive translation unstable case. [doris]
morningman merged PR #46385: URL: https://github.com/apache/doris/pull/46385 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compile) fix fe compile failed by code too large [doris]
github-actions[bot] commented on PR #46408: URL: https://github.com/apache/doris/pull/46408#issuecomment-2569327398 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compile) fix fe compile failed by code too large [doris]
github-actions[bot] commented on PR #46408: URL: https://github.com/apache/doris/pull/46408#issuecomment-2569327361 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression)fix hive translation unstable case. [doris]
github-actions[bot] commented on PR #46385: URL: https://github.com/apache/doris/pull/46385#issuecomment-2569323511 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-2.1: [fix](regression)fix hive translation unstable case. #46385 [doris]
github-actions[bot] opened a new pull request, #46409: URL: https://github.com/apache/doris/pull/46409 Cherry-picked from #46385 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](regression)fix hive translation unstable case. (#46385)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3c9b29a0bb5 [fix](regression)fix hive translation unstable case.
(#46385)
3c9b29a0bb5 is described below
commit 3c9b29a0bb598ca7ffe32fc64fa918fb06bbf236
Author: daidai
AuthorDate: Fri Jan 3 22:39:09 2025 +0800
[fix](regression)fix hive translation unstable case. (#46385)
### What problem does this PR solve?
Problem Summary:
just fix case.
---
.../hive/scripts/create_preinstalled_scripts/run25.hql | 7 ---
.../hive/test_hive_translation_insert_only.groovy | 1 +
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git
a/docker/thirdparties/docker-compose/hive/scripts/create_preinstalled_scripts/run25.hql
b/docker/thirdparties/docker-compose/hive/scripts/create_preinstalled_scripts/run25.hql
index da6400bdff0..66e73f51df8 100755
---
a/docker/thirdparties/docker-compose/hive/scripts/create_preinstalled_scripts/run25.hql
+++
b/docker/thirdparties/docker-compose/hive/scripts/create_preinstalled_scripts/run25.hql
@@ -61,7 +61,8 @@ STORED AS ORC;
INSERT INTO TABLE orc_to_acid_compacted_tb PARTITION (part_col=101) VALUES (1,
'A'), (3, 'C');
INSERT INTO TABLE orc_to_acid_compacted_tb PARTITION (part_col=102) VALUES (2,
'B');
ALTER TABLE orc_to_acid_compacted_tb SET TBLPROPERTIES
('transactional'='true');
-ALTER TABLE orc_to_acid_compacted_tb COMPACT 'major';
+ALTER TABLE orc_to_acid_compacted_tb partition(part_col='101') COMPACT 'major'
and wait;
+ALTER TABLE orc_to_acid_compacted_tb partition(part_col='102') COMPACT 'major'
and wait;
INSERT INTO TABLE orc_to_acid_compacted_tb PARTITION (part_col=102) VALUES (4,
'D');
update orc_to_acid_compacted_tb set value = "CC" where id = 3;
update orc_to_acid_compacted_tb set value = "BB" where id = 2;
@@ -75,7 +76,7 @@ insert into orc_acid_minor values (1, 'A');
insert into orc_acid_minor values (2, 'B');
insert into orc_acid_minor values (3, 'C');
update orc_acid_minor set value = "BB" where id = 2;
-ALTER TABLE orc_acid_minor COMPACT 'minor';
+ALTER TABLE orc_acid_minor COMPACT 'minor' and wait;
insert into orc_acid_minor values (4, 'D');
update orc_acid_minor set value = "DD" where id = 4;
DELETE FROM orc_acid_minor WHERE id = 3;
@@ -89,7 +90,7 @@ insert into orc_acid_major values (1, 'A');
insert into orc_acid_major values (2, 'B');
insert into orc_acid_major values (3, 'C');
update orc_acid_major set value = "BB" where id = 2;
-ALTER TABLE orc_acid_major COMPACT 'minor';
+ALTER TABLE orc_acid_major COMPACT 'minor' and wait;
insert into orc_acid_major values (4, 'D');
update orc_acid_major set value = "DD" where id = 4;
DELETE FROM orc_acid_major WHERE id = 3;
diff --git
a/regression-test/suites/external_table_p2/hive/test_hive_translation_insert_only.groovy
b/regression-test/suites/external_table_p2/hive/test_hive_translation_insert_only.groovy
index 62a9727ed86..758417c3237 100644
---
a/regression-test/suites/external_table_p2/hive/test_hive_translation_insert_only.groovy
+++
b/regression-test/suites/external_table_p2/hive/test_hive_translation_insert_only.groovy
@@ -21,6 +21,7 @@ suite("test_hive_translation_insert_only",
"p2,external,hive,external_remote,ext
//hudi hive use same catalog in p2.
if (enabled == null || !enabled.equalsIgnoreCase("true")) {
logger.info("disable test")
+return;
}
String props = context.config.otherConfigs.get("hudiEmrCatalog")
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [fix](regression)fix hive translation unstable case. #46385 [doris]
dataroaring closed pull request #46409: branch-2.1: [fix](regression)fix hive translation unstable case. #46385 URL: https://github.com/apache/doris/pull/46409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [fix](regression)fix hive translation unstable case. #46385 [doris]
Thearas commented on PR #46409: URL: https://github.com/apache/doris/pull/46409#issuecomment-2569329828 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-46385-branch-2.1 updated (e9921d7a076 -> 2086f69d2e3)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-46385-branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from e9921d7a076 branch-2.1: [fix](external catalog) Fixed non-master FE log replay during refresh (#46373) add 2086f69d2e3 [fix](regression)fix hive translation unstable case. (#46385) No new revisions were added by this update. Summary of changes: .../hive/scripts/create_preinstalled_scripts/run25.hql | 7 --- .../hive/test_hive_translation_insert_only.groovy | 1 + 2 files changed, 5 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [fix](regression)fix hive translation unstable case. #46385 [doris]
Thearas commented on PR #46409: URL: https://github.com/apache/doris/pull/46409#issuecomment-2569329908 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-46385-branch-3.0 deleted (was b57dc79cc96)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-46385-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git was b57dc79cc96 branch-3.0: [Fix](index build) should not append index to meta while column id is -1 #46307 (#46335) The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-46385-branch-3.0 created (now b57dc79cc96)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-46385-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at b57dc79cc96 branch-3.0: [Fix](index build) should not append index to meta while column id is -1 #46307 (#46335) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](Schema Change)light schema change support add key column [doris]
mongo360 commented on PR #42911: URL: https://github.com/apache/doris/pull/42911#issuecomment-2569346354 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-46385-branch-2.1 created (now e9921d7a076)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-46385-branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git at e9921d7a076 branch-2.1: [fix](external catalog) Fixed non-master FE log replay during refresh (#46373) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mem) heap-buffer-overflow for function convert_to [doris]
doris-robot commented on PR #46405: URL: https://github.com/apache/doris/pull/46405#issuecomment-2569352323 TeamCity be ut coverage result: Function Coverage: 38.88% (10127/26048) Line Coverage: 29.92% (85701/286405) Region Coverage: 29.02% (43717/150670) Branch Coverage: 25.56% (22315/87316) Coverage Report: http://coverage.selectdb-in.cc/coverage/d940ac512a56cea8223fe39520ffeb433ecb1417_d940ac512a56cea8223fe39520ffeb433ecb1417/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](conf) Set enable_advance_next_id=true by default [doris]
gavinchou merged PR #44790: URL: https://github.com/apache/doris/pull/44790 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (7b45bc3d8ba -> 10eac2257d1)
This is an automated email from the ASF dual-hosted git repository. gavinchou pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 7b45bc3d8ba [fix](nereids)fix rangeSet intersect in PartitionPredicateToRange (#46087) add 10eac2257d1 [chore](conf) Set enable_advance_next_id=true by default (#44790) No new revisions were added by this update. Summary of changes: fe/fe-common/src/main/java/org/apache/doris/common/Config.java| 2 +- .../org/apache/doris/analysis/ExportToOutfileLogicalPlanTest.java | 8 .../java/org/apache/doris/common/proc/IndexesProcNodeTest.java| 2 +- .../java/org/apache/doris/service/FrontendServiceImplTest.java| 7 +-- .../src/test/java/org/apache/doris/utframe/TestWithFeService.java | 4 5 files changed, 19 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](spill) spill and reserve [doris]
doris-robot commented on PR #46230: URL: https://github.com/apache/doris/pull/46230#issuecomment-2569065105 ClickBench: Total hot run time: 31.36 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 393f12ea6ad950b270425928c4ce386f2d2623eb, data reload: false query1 0.030.030.03 query2 0.070.030.04 query3 0.240.070.07 query4 1.620.110.11 query5 0.440.400.39 query6 1.150.650.65 query7 0.020.020.02 query8 0.040.030.03 query9 0.580.510.52 query10 0.550.570.55 query11 0.140.110.11 query12 0.130.110.11 query13 0.610.610.59 query14 2.852.772.90 query15 0.900.830.82 query16 0.380.370.38 query17 1.021.031.06 query18 0.220.200.20 query19 1.881.812.00 query20 0.010.010.01 query21 15.36 0.930.59 query22 0.740.870.67 query23 15.27 1.470.60 query24 3.010.871.62 query25 0.180.110.07 query26 0.380.140.14 query27 0.070.050.04 query28 13.90 1.531.04 query29 12.57 3.913.26 query30 0.250.090.07 query31 2.800.580.38 query32 3.250.540.47 query33 3.053.113.19 query34 16.69 5.144.56 query35 4.524.504.51 query36 0.620.480.49 query37 0.090.060.06 query38 0.040.030.04 query39 0.040.030.02 query40 0.160.140.13 query41 0.080.030.02 query42 0.040.020.03 query43 0.040.030.02 Total cold run time: 106.03 s Total hot run time: 31.36 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) insert into with cte cannot be explained [doris]
doris-robot commented on PR #46384: URL: https://github.com/apache/doris/pull/46384#issuecomment-2569064964 TPC-DS: Total hot run time: 195721 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 7f01c1506bd14ce0d6602c7decc4c8b256a65cac, data reload: false query1 1288966 900 900 query2 6419229123402291 query3 10904 471844784478 query4 33269 23917 23163 23163 query5 3703613 441 441 query6 263 195 187 187 query7 3981498 299 299 query8 291 255 247 247 query9 9349264226172617 query10 461 317 238 238 query11 17829 15310 15012 15012 query12 157 102 98 98 query13 1561553 390 390 query14 9157667171706671 query15 271 222 208 208 query16 7929620 457 457 query17 1509758 586 586 query18 2010447 314 314 query19 217 185 157 157 query20 145 114 114 114 query21 207 129 112 112 query22 4637482846544654 query23 34454 33834 33729 33729 query24 6430231123182311 query25 444 455 413 413 query26 817 288 164 164 query27 2128466 350 350 query28 5576246024412441 query29 616 568 456 456 query30 218 185 154 154 query31 998 922 837 837 query32 70 62 59 59 query33 478 367 309 309 query34 781 827 520 520 query35 837 856 770 770 query36 10251042983 983 query37 117 118 80 80 query38 4261416942664169 query39 1500148315001483 query40 210 116 101 101 query41 45 45 44 44 query42 124 107 102 102 query43 492 542 496 496 query44 1289819 818 818 query45 180 171 173 171 query46 895 1066655 655 query47 1975199419281928 query48 384 406 323 323 query49 715 504 419 419 query50 643 659 390 390 query51 7116728272557255 query52 103 99 91 91 query53 235 256 190 190 query54 485 492 398 398 query55 85 78 74 74 query56 255 249 248 248 query57 1234122011941194 query58 235 221 219 219 query59 3120331631073107 query60 282 261 249 249 query61 111 103 107 103 query62 906 807 769 769 query63 231 187 193 187 query64 30811010668 668 query65 3350328632683268 query66 955 407 302 302 query67 16202 15872 15483 15483 query68 7946694 523 523 query69 490 297 244 244 query70 1199107711401077 query71 430 276 245 245 query72 6489386038193819 query73 643 759 354 354 query74 10033 905290459045 query75 4032311526382638 query76 36311175760 760 query77 755 364 273 273 query78 10131 10007 94899489 query79 3931812 576 576 query80 724 510 417 417 query81 464 360 222 222 query82 576 149 123 123 query83 192 160 149 149 query84 279 86 74 74 query85 783 373 306 306 query86 350 319 308 308 query87 4696433944204339 query88 4534216721612161 query89 428 322 305 305 query90 1865186 187 186 query91 135 133 106 106 query92 68 54 54 54 query93 2609875 533 533 query94 649 398 279 279 query95 331 264 250 250 query96 483 608 277 277 query97 2899301428432843 query98 212 205 200 200 query99 1736156714321432 Total cold run time: 294968 ms Total hot run time: 195721 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [fix](schema-change) Fix job replay failure when partitions added to table after job finish (#46166) [doris]
dataroaring merged PR #46367: URL: https://github.com/apache/doris/pull/46367 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](schema-change) Fix job replay failure when partitions added to table after job finish (#46166) (#46367)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new c2a17b8ad9b [fix](schema-change) Fix job replay failure when
partitions added to table after job finish (#46166) (#46367)
c2a17b8ad9b is described below
commit c2a17b8ad9b5975db2e73d4d2a74a953ba05c547
Author: Siyang Tang
AuthorDate: Fri Jan 3 19:25:43 2025 +0800
[fix](schema-change) Fix job replay failure when partitions added to table
after job finish (#46166) (#46367)
Cherry-picked from #46166
---
.../org/apache/doris/alter/SchemaChangeJobV2.java| 20 ++--
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
index 7bf6925237d..e98d8066a11 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
@@ -586,20 +586,20 @@ public class SchemaChangeJobV2 extends AlterJobV2 {
} // end for partitions
// all partitions are good
onFinished(tbl);
-} finally {
-tbl.writeUnlock();
-}
-pruneMeta();
+pruneMeta();
-LOG.info("schema change job finished: {}", jobId);
+LOG.info("schema change job finished: {}", jobId);
-changeTableState(dbId, tableId, OlapTableState.NORMAL);
-LOG.info("set table's state to NORMAL, table id: {}, job id: {}",
tableId, jobId);
+changeTableState(dbId, tableId, OlapTableState.NORMAL);
+LOG.info("set table's state to NORMAL, table id: {}, job id: {}",
tableId, jobId);
-this.jobState = JobState.FINISHED;
-this.finishedTimeMs = System.currentTimeMillis();
-Env.getCurrentEnv().getEditLog().logAlterJob(this);
+this.jobState = JobState.FINISHED;
+this.finishedTimeMs = System.currentTimeMillis();
+Env.getCurrentEnv().getEditLog().logAlterJob(this);
+} finally {
+tbl.writeUnlock();
+}
// Drop table column stats after schema change finished.
Env.getCurrentEnv().getAnalysisManager().dropStats(tbl);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](case) fix cluster key case in cloud mode [doris]
github-actions[bot] commented on PR #46348: URL: https://github.com/apache/doris/pull/46348#issuecomment-2569090272 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](case) fix cluster key case in cloud mode [doris]
github-actions[bot] commented on PR #46348: URL: https://github.com/apache/doris/pull/46348#issuecomment-2569090300 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [opt](nereids) improve prune partition with lots of `in (xxx)` #46261 [doris]
morrySnow merged PR #46285: URL: https://github.com/apache/doris/pull/46285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: branch-3.0: [opt](nereids) improve prune partition with lots of `in (xxx)` #46261 (#46285)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 04b18d9dcb6 branch-3.0: [opt](nereids) improve prune partition with
lots of `in (xxx)` #46261 (#46285)
04b18d9dcb6 is described below
commit 04b18d9dcb69bbc0a6a5692667cfe57f0444cd54
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Jan 3 19:38:11 2025 +0800
branch-3.0: [opt](nereids) improve prune partition with lots of `in (xxx)`
#46261 (#46285)
Cherry-picked from #46261
Co-authored-by: 924060929
---
.../rules/expression/rules/OneRangePartitionEvaluator.java | 9 +
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/OneRangePartitionEvaluator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/OneRangePartitionEvaluator.java
index 7e91d5502f7..0a9d0bbed01 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/OneRangePartitionEvaluator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/OneRangePartitionEvaluator.java
@@ -62,6 +62,8 @@ import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.TreeRangeSet;
import java.util.ArrayList;
import java.util.HashMap;
@@ -362,13 +364,12 @@ public class OneRangePartitionEvaluator
if (exprRanges.containsKey(inPredicate.getCompareExpr())
&&
inPredicate.getOptions().stream().allMatch(Literal.class::isInstance)) {
Expression compareExpr = inPredicate.getCompareExpr();
-ColumnRange unionLiteralRange = ColumnRange.empty();
ColumnRange compareExprRange =
result.childrenResult.get(0).columnRanges.get(compareExpr);
+RangeSet union = TreeRangeSet.create();
for (Expression expr : inPredicate.getOptions()) {
-unionLiteralRange = unionLiteralRange.union(
-
compareExprRange.intersect(ColumnRange.singleton((Literal) expr)));
+
union.addAll(compareExprRange.intersect(ColumnRange.singleton((Literal)
expr)).asRanges());
}
-result = intersectSlotRange(result, exprRanges, compareExpr,
unionLiteralRange);
+result = intersectSlotRange(result, exprRanges, compareExpr, new
ColumnRange(union));
}
result = result.withRejectNot(false);
return result;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-46261-branch-3.0 deleted (was eaa8e5f9247)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch auto-pick-46261-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git was eaa8e5f9247 [opt](nereids) improve prune partition with lots of `in (xxx)` (#46261) The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](coordinator) optimize parallel degree of shuffle when use nereids [doris]
924060929 commented on PR #44754: URL: https://github.com/apache/doris/pull/44754#issuecomment-2569148739 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) fix bug of LogicalRepeat computing uniform datatrait [doris]
feiniaofeiafei commented on PR #46352: URL: https://github.com/apache/doris/pull/46352#issuecomment-2569154461 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](recycler) Fix recycler Hdfs accessor list all bug [doris]
Yukang-Lian commented on PR #46393: URL: https://github.com/apache/doris/pull/46393#issuecomment-2569158499 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](fe.conf) use g1 by default for jdk17 [doris]
dataroaring opened a new pull request, #46395: URL: https://github.com/apache/doris/pull/46395 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](fe.conf) use g1 by default for jdk17 [doris]
dataroaring commented on PR #46395: URL: https://github.com/apache/doris/pull/46395#issuecomment-2569165797 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](fe.conf) use g1 by default for jdk17 [doris]
Thearas commented on PR #46395: URL: https://github.com/apache/doris/pull/46395#issuecomment-2569165405 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](iceberg)support iceberg in batch mode [doris]
Thearas commented on PR #46398: URL: https://github.com/apache/doris/pull/46398#issuecomment-2569182112 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](azure) let backup to azure work and use https for azure [doris]
doris-robot commented on PR #46064: URL: https://github.com/apache/doris/pull/46064#issuecomment-2569195422 ClickBench: Total hot run time: 31.44 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 83925b38bd1cfbaf64cd5f20b816de8b0a7b1ba2, data reload: false query1 0.040.030.03 query2 0.080.040.03 query3 0.240.070.07 query4 1.640.100.11 query5 0.420.420.44 query6 1.140.650.65 query7 0.020.020.02 query8 0.040.030.03 query9 0.600.500.51 query10 0.550.570.56 query11 0.140.100.10 query12 0.140.110.10 query13 0.610.610.60 query14 2.722.842.86 query15 0.890.820.82 query16 0.360.380.38 query17 0.980.991.05 query18 0.230.210.21 query19 1.871.751.97 query20 0.010.010.01 query21 15.37 0.960.59 query22 0.750.930.63 query23 15.22 1.350.61 query24 2.961.051.12 query25 0.230.230.11 query26 0.300.150.13 query27 0.070.070.07 query28 13.60 1.541.04 query29 12.54 3.983.26 query30 0.250.090.07 query31 2.830.590.39 query32 3.230.540.46 query33 3.073.053.09 query34 16.86 5.154.51 query35 4.514.474.47 query36 0.650.490.48 query37 0.090.060.06 query38 0.050.040.03 query39 0.040.030.02 query40 0.160.140.14 query41 0.080.020.02 query42 0.040.020.02 query43 0.030.020.03 Total cold run time: 105.65 s Total hot run time: 31.44 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](external)using saved member variables for 2.1 (#46156) [doris]
wuwenchi opened a new pull request, #46400: URL: https://github.com/apache/doris/pull/46400 bp: #46156 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](external)using saved member variables for 2.1 (#46156) [doris]
Thearas commented on PR #46400: URL: https://github.com/apache/doris/pull/46400#issuecomment-2569197385 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](coordinator) optimize parallel degree of shuffle when use nereids [doris]
doris-robot commented on PR #44754: URL: https://github.com/apache/doris/pull/44754#issuecomment-2569198869 TPC-H: Total hot run time: 30672 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 7b91be887838d561af39baa923e397fb3359715c, data reload: false -- Round 1 -- q1 17568 609560356035 q2 2048288 162 162 q3 10449 1205761 761 q4 10205 852 427 427 q5 7512216119671967 q6 206 182 146 146 q7 895 733 611 611 q8 9247130811231123 q9 3984306229802980 q10 6786232818861886 q11 483 285 253 253 q12 348 360 225 225 q13 17782 372330713071 q14 235 235 224 224 q15 561 502 498 498 q16 639 618 568 568 q17 564 856 337 337 q18 6942641964606419 q19 1770951 509 509 q20 295 306 178 178 q21 2811215319861986 q22 345 329 306 306 Total cold run time: 101675 ms Total hot run time: 30672 ms - Round 2, with runtime_filter_mode=off - q1 6436618461886184 q2 227 322 229 229 q3 2244262623302330 q4 1364181713371337 q5 4345475146354635 q6 184 171 136 136 q7 1948187218081808 q8 2576281627202720 q9 5107496350164963 q10 3072335428542854 q11 588 527 498 498 q12 676 799 619 619 q13 3507379132043204 q14 295 314 286 286 q15 562 513 505 505 q16 667 697 675 675 q17 1209171512641264 q18 7593741874207418 q19 856 115010811081 q20 1946203619171917 q21 5785527150595059 q22 619 628 574 574 Total cold run time: 51806 ms Total hot run time: 50296 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: [enhancement](cloud) reconnect after the RPC request to the meta service fails (#45668) (#46358)
This is an automated email from the ASF dual-hosted git repository.
gavinchou pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 920c89c3f4a [enhancement](cloud) reconnect after the RPC request to
the meta service fails (#45668) (#46358)
920c89c3f4a is described below
commit 920c89c3f4a0f79b71647b691d8e95a69553b037
Author: Luwei
AuthorDate: Fri Jan 3 21:11:18 2025 +0800
[enhancement](cloud) reconnect after the RPC request to the meta service
fails (#45668) (#46358)
pick #45668
-
Co-authored-by: Gavin Chou
---
be/src/cloud/cloud_meta_mgr.cpp| 191 ++---
be/src/cloud/config.cpp| 4 +
be/src/cloud/config.h | 4 +
.../main/java/org/apache/doris/common/Config.java | 4 +
.../apache/doris/cloud/rpc/MetaServiceProxy.java | 52 +-
5 files changed, 191 insertions(+), 64 deletions(-)
diff --git a/be/src/cloud/cloud_meta_mgr.cpp b/be/src/cloud/cloud_meta_mgr.cpp
index 55ffe46286c..747baea780c 100644
--- a/be/src/cloud/cloud_meta_mgr.cpp
+++ b/be/src/cloud/cloud_meta_mgr.cpp
@@ -139,7 +139,71 @@ class MetaServiceProxy {
public:
static Status get_client(std::shared_ptr* stub) {
TEST_SYNC_POINT_RETURN_WITH_VALUE("MetaServiceProxy::get_client",
Status::OK(), stub);
-return get_pooled_client(stub);
+return get_pooled_client(stub, nullptr);
+}
+
+static Status get_proxy(MetaServiceProxy** proxy) {
+// The 'stub' is a useless parameter, added only to reuse the
`get_pooled_client` function.
+std::shared_ptr stub;
+return get_pooled_client(&stub, proxy);
+}
+
+void set_unhealthy() {
+std::unique_lock lock(_mutex);
+maybe_unhealthy = true;
+}
+
+bool need_reconn(long now) {
+return maybe_unhealthy && ((now - last_reconn_time_ms.front()) >
+
config::meta_service_rpc_reconnect_interval_ms);
+}
+
+Status get(std::shared_ptr* stub) {
+using namespace std::chrono;
+
+auto now =
duration_cast(system_clock::now().time_since_epoch()).count();
+{
+std::shared_lock lock(_mutex);
+if (_deadline_ms >= now && !is_idle_timeout(now) &&
!need_reconn(now)) {
+_last_access_at_ms.store(now, std::memory_order_relaxed);
+*stub = _stub;
+return Status::OK();
+}
+}
+
+auto channel = std::make_unique();
+Status s = init_channel(channel.get());
+if (!s.ok()) [[unlikely]] {
+return s;
+}
+
+*stub = std::make_shared(channel.release(),
+
google::protobuf::Service::STUB_OWNS_CHANNEL);
+
+long deadline = now;
+// connection age only works without list endpoint.
+if (!is_meta_service_endpoint_list() &&
+config::meta_service_connection_age_base_seconds > 0) {
+std::default_random_engine rng(static_cast(now));
+std::uniform_int_distribution<> uni(
+config::meta_service_connection_age_base_seconds,
+config::meta_service_connection_age_base_seconds * 2);
+deadline = now +
duration_cast(seconds(uni(rng))).count();
+} else {
+deadline = LONG_MAX;
+}
+
+// Last one WIN
+std::unique_lock lock(_mutex);
+_last_access_at_ms.store(now, std::memory_order_relaxed);
+_deadline_ms = deadline;
+_stub = *stub;
+
+last_reconn_time_ms.push(now);
+last_reconn_time_ms.pop();
+maybe_unhealthy = false;
+
+return Status::OK();
}
private:
@@ -147,7 +211,17 @@ private:
return config::meta_service_endpoint.find(',') != std::string::npos;
}
-static Status get_pooled_client(std::shared_ptr* stub) {
+/**
+* This function initializes a pool of `MetaServiceProxy` objects and
selects one using
+* round-robin. It returns a client stub via the selected proxy.
+*
+* @param stub A pointer to a shared pointer of `MetaService_Stub` to be
retrieved.
+* @param proxy (Optional) A pointer to store the selected
`MetaServiceProxy`.
+*
+* @return Status Returns `Status::OK()` on success or an error status on
failure.
+*/
+static Status get_pooled_client(std::shared_ptr* stub,
+MetaServiceProxy** proxy) {
static std::once_flag proxies_flag;
static size_t num_proxies = 1;
static std::atomic index(0);
@@ -164,10 +238,16 @@ private:
for (size_t i = 0; i + 1 < num_proxies; ++i) {
size_t next_index = index.fetch_add(1, std::memory_order_relaxed)
% num_proxies;
Status s = proxies[next_index].get(stub)
Re: [PR] [enhancement](cloud) reconnect after the RPC request to the meta service fails (#45668) [doris]
gavinchou merged PR #46358: URL: https://github.com/apache/doris/pull/46358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) let Physical plan compute logicalProperties.datatrait [doris]
hello-stephen commented on PR #46401: URL: https://github.com/apache/doris/pull/46401#issuecomment-2569203672 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [opt](nereids) let Physical plan compute logicalProperties.datatrait [doris]
englefly opened a new pull request, #46401: URL: https://github.com/apache/doris/pull/46401 ### What problem does this PR solve? logicalProperties.datatrait is not only used by LogicalPlan, it is also used in postprocessor, especially used in RuntimeFilterGeneration. T1 join T2 on T1.a=T2.a where T1.a=1 and T2.a=1 Because both T1.a and T2.a are unique value "1", the RF: T1.a->T2.a is useless Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) fix bug of LogicalRepeat computing uniform datatrait [doris]
doris-robot commented on PR #46352: URL: https://github.com/apache/doris/pull/46352#issuecomment-2569204641 TPC-H: Total hot run time: 32482 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit f26b6717fa4e8031d30b8df11336bfb9d3902226, data reload: false -- Round 1 -- q1 17598 608760036003 q2 2183301 176 176 q3 10531 1360786 786 q4 10278 872 435 435 q5 7521217219911991 q6 206 176 146 146 q7 906 774 601 601 q8 9242136011551155 q9 5258484449274844 q10 6731230618601860 q11 470 283 262 262 q12 336 359 215 215 q13 17761 359930883088 q14 226 229 219 219 q15 547 515 504 504 q16 630 607 592 592 q17 563 842 327 327 q18 6767631263066306 q19 2201969 548 548 q20 312 311 182 182 q21 2867220719481948 q22 355 341 294 294 Total cold run time: 103489 ms Total hot run time: 32482 ms - Round 2, with runtime_filter_mode=off - q1 6306616862006168 q2 235 319 231 231 q3 2274265222992299 q4 1429183113401340 q5 4376472947554729 q6 183 178 145 145 q7 2038193218321832 q8 2626280726382638 q9 7291715273147152 q10 3084333628642864 q11 589 538 488 488 q12 691 731 564 564 q13 3557384432483248 q14 288 332 285 285 q15 571 508 515 508 q16 654 683 658 658 q17 1217171512581258 q18 7545743873777377 q19 829 951 1148951 q20 2030203919451945 q21 5652527749164916 q22 596 611 573 573 Total cold run time: 54061 ms Total hot run time: 52169 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](external)using saved member variables for 2.1 (#46156) [doris]
wuwenchi commented on PR #46400: URL: https://github.com/apache/doris/pull/46400#issuecomment-2569197624 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap [doris]
englefly closed pull request #46224: [fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap URL: https://github.com/apache/doris/pull/46224 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch asf-site updated (30d71764f0e -> 778f4dd7897)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard 30d71764f0e Automated deployment with doris branch @ 57c7af92112c4f02a887cbf864c4a1b641c3cc56 new 778f4dd7897 Automated deployment with doris branch @ 57c7af92112c4f02a887cbf864c4a1b641c3cc56 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (30d71764f0e) \ N -- N -- N refs/heads/asf-site (778f4dd7897) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/1.2/search-index.json | 2 +- docs/2.0/search-index.json | 2 +- docs/3.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/1.2/search-index.json | 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/3.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 10 files changed, 10 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch asf-site updated (fc1c0c293ec -> 73b6a5ab130)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard fc1c0c293ec Automated deployment with doris branch @ 8512da5c344b974360b07a303308f7edf1b5350e new 73b6a5ab130 Automated deployment with doris branch @ 8512da5c344b974360b07a303308f7edf1b5350e This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (fc1c0c293ec) \ N -- N -- N refs/heads/asf-site (73b6a5ab130) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/1.2/search-index.json | 2 +- docs/2.0/search-index.json | 2 +- docs/3.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/1.2/search-index.json | 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/3.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 10 files changed, 10 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch asf-site updated (73b6a5ab130 -> bdd7cb55f62)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard 73b6a5ab130 Automated deployment with doris branch @ 8512da5c344b974360b07a303308f7edf1b5350e new bdd7cb55f62 Automated deployment with doris branch @ 8512da5c344b974360b07a303308f7edf1b5350e This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (73b6a5ab130) \ N -- N -- N refs/heads/asf-site (bdd7cb55f62) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/1.2/search-index.json | 2 +- docs/2.0/search-index.json | 2 +- docs/3.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/1.2/search-index.json | 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/3.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 10 files changed, 10 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap-branch3.0 [doris]
englefly commented on PR #46224: URL: https://github.com/apache/doris/pull/46224#issuecomment-2569922446 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap-branch3.0 [doris]
englefly commented on PR #46224: URL: https://github.com/apache/doris/pull/46224#issuecomment-2569922367 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap [doris]
englefly commented on PR #46406: URL: https://github.com/apache/doris/pull/46406#issuecomment-2569920826 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap (branch-2.1) [doris]
englefly commented on PR #46407: URL: https://github.com/apache/doris/pull/46407#issuecomment-2569921896 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](cloud) reconnect after the RPC request to the meta service fails (#45668) [doris]
luwei16 commented on PR #46358: URL: https://github.com/apache/doris/pull/46358#issuecomment-2568854030 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](mtmv) Chose mv more easier when mv on external table [doris]
doris-robot commented on PR #46368: URL: https://github.com/apache/doris/pull/46368#issuecomment-2568853852 ClickBench: Total hot run time: 31.08 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 0ac0def59dfb1cd6bef350666ac78a5a185f63f9, data reload: false query1 0.040.030.03 query2 0.070.030.03 query3 0.230.070.07 query4 1.610.110.10 query5 0.430.390.44 query6 1.170.660.65 query7 0.020.010.02 query8 0.040.040.03 query9 0.590.480.51 query10 0.560.570.54 query11 0.140.110.10 query12 0.140.110.11 query13 0.600.610.60 query14 2.742.752.82 query15 0.910.820.82 query16 0.400.390.36 query17 1.061.051.07 query18 0.240.210.20 query19 2.001.941.86 query20 0.020.020.01 query21 15.61 0.910.57 query22 0.750.810.81 query23 15.12 1.440.52 query24 3.291.160.72 query25 0.240.190.08 query26 0.260.140.13 query27 0.070.060.05 query28 13.80 1.491.04 query29 12.57 3.933.26 query30 0.240.080.07 query31 2.830.630.38 query32 3.220.540.45 query33 3.263.083.19 query34 16.79 5.164.47 query35 4.494.484.47 query36 0.650.480.47 query37 0.090.060.06 query38 0.050.040.03 query39 0.040.020.02 query40 0.170.130.13 query41 0.080.030.03 query42 0.040.020.02 query43 0.040.030.03 Total cold run time: 106.71 s Total hot run time: 31.08 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](arrow-flight-sql) Fix arrow-flight-sql ConnectContext to use a unified ID [doris]
github-actions[bot] commented on PR #46284: URL: https://github.com/apache/doris/pull/46284#issuecomment-2568856039 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch asf-site updated (85d7335a0bc -> 773d6d93061)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard 85d7335a0bc Automated deployment with doris branch @ f37963f1d555335bf490f6010900d44b2487cbde new 773d6d93061 Automated deployment with doris branch @ f37963f1d555335bf490f6010900d44b2487cbde This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (85d7335a0bc) \ N -- N -- N refs/heads/asf-site (773d6d93061) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/1.2/search-index.json | 2 +- docs/2.0/search-index.json | 2 +- docs/3.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/1.2/search-index.json | 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/3.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 10 files changed, 10 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](arrow-flight-sql) Fix arrow-flight-sql ConnectContext to use a unified ID [doris]
github-actions[bot] commented on PR #46284: URL: https://github.com/apache/doris/pull/46284#issuecomment-2568855997 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](external catalog) Fixed non-master FE log replay during refresh [doris]
github-actions[bot] commented on PR #46369: URL: https://github.com/apache/doris/pull/46369#issuecomment-2568856375 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](external catalog) Fixed non-master FE log replay during refresh [doris]
github-actions[bot] commented on PR #46369: URL: https://github.com/apache/doris/pull/46369#issuecomment-2568856419 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](parquet) parse dict page in ColumnChunkReader::init [doris]
suxiaogang223 opened a new pull request, #46372: URL: https://github.com/apache/doris/pull/46372 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #45740 Problem Summary: This pull request includes several changes to the `ColumnChunkReader` class and its methods in the file `be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp`. The main focus is on improving the handling of dictionary pages and simplifying the code. Improvements to dictionary page handling: * [`ColumnChunkReader::init()`](diffhunk://#diff-97f77661f2bd0292ee695640daa792b52bd58f04a026929e2ee6a9ded0f53977L74-R75): Re-enabled the dictionary page parsing by uncommenting the calls to `_page_reader->next_page_header()` and `_decode_dict_page()`. * [`ColumnChunkReader::next_page()`](diffhunk://#diff-97f77661f2bd0292ee695640daa792b52bd58f04a026929e2ee6a9ded0f53977L96-L109): Removed the logic for checking and decoding dictionary pages, as this is now handled in the `init()` method. Code simplification: * [`ColumnChunkReader::load_page_data()`](diffhunk://#diff-97f77661f2bd0292ee695640daa792b52bd58f04a026929e2ee6a9ded0f53977R120-R121): Added a DCHECK to ensure that the page header is not a dictionary page. * `ColumnChunkReader` class: Removed the `_dict_checked` member variable, as it is no longer needed. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) parse dict page in ColumnChunkReader::init [doris]
Thearas commented on PR #46372: URL: https://github.com/apache/doris/pull/46372#issuecomment-2568858401 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cancel) Fix `BufferControlBlock` cancel msg [doris]
github-actions[bot] commented on PR #46266: URL: https://github.com/apache/doris/pull/46266#issuecomment-2568860881 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: branch-2.1: [improvement](http) improve table schema api for catalog table #45933 (#45982)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new ff68859546e branch-2.1: [improvement](http) improve table schema api
for catalog table #45933 (#45982)
ff68859546e is described below
commit ff68859546eadde9e20d8120c953bfbc83edd8c7
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Jan 3 16:40:12 2025 +0800
branch-2.1: [improvement](http) improve table schema api for catalog table
#45933 (#45982)
Cherry-picked from #45933
Co-authored-by: wudi
---
.../doris/httpv2/controller/BaseController.java| 10 +++-
.../doris/httpv2/rest/RestBaseController.java | 1 +
.../doris/httpv2/rest/TableSchemaAction.java | 39 ++
.../http_rest_api/get/test_schema_api.groovy | 60 ++
4 files changed, 98 insertions(+), 12 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/controller/BaseController.java
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/controller/BaseController.java
index 026b1f8bda3..60fc9258717 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/controller/BaseController.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/controller/BaseController.java
@@ -117,7 +117,7 @@ public class BaseController {
}
private ActionAuthorizationInfo checkCookie(HttpServletRequest request,
HttpServletResponse response,
-boolean checkAuth) {
+boolean checkAuth) {
List sessionIds = getCookieValues(request, PALO_SESSION_ID,
response);
if (sessionIds.isEmpty()) {
return null;
@@ -218,8 +218,14 @@ public class BaseController {
protected void checkTblAuth(UserIdentity currentUser, String db, String
tbl, PrivPredicate predicate)
throws UnauthorizedException {
+checkTblAuth(currentUser, InternalCatalog.INTERNAL_CATALOG_NAME, db,
tbl, predicate);
+}
+
+protected void checkTblAuth(UserIdentity currentUser, String catalog,
String db, String tbl,
+PrivPredicate predicate)
+throws UnauthorizedException {
if (!Env.getCurrentEnv().getAccessManager()
-.checkTblPriv(currentUser,
InternalCatalog.INTERNAL_CATALOG_NAME, db, tbl, predicate)) {
+.checkTblPriv(currentUser, catalog, db, tbl, predicate)) {
throw new UnauthorizedException("Access denied; you need (at least
one of) the "
+ predicate.getPrivs().toString() + " privilege(s) for
this operation");
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/RestBaseController.java
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/RestBaseController.java
index 5621ffaccde..bf07eda4f8d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/RestBaseController.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/RestBaseController.java
@@ -49,6 +49,7 @@ import javax.servlet.http.HttpServletResponse;
public class RestBaseController extends BaseController {
protected static final String NS_KEY = "ns";
+protected static final String CATALOG_KEY = "catalog";
protected static final String DB_KEY = "db";
protected static final String TABLE_KEY = "table";
protected static final String LABEL_KEY = "label";
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/TableSchemaAction.java
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/TableSchemaAction.java
index ae1b3956179..0f362b4aad8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/TableSchemaAction.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/TableSchemaAction.java
@@ -19,20 +19,26 @@ package org.apache.doris.httpv2.rest;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.DatabaseIf;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.catalog.ScalarType;
import org.apache.doris.catalog.Table;
+import org.apache.doris.catalog.TableIf;
import org.apache.doris.catalog.Type;
+import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.DorisHttpException;
import org.apache.doris.common.MetaNotFoundException;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.httpv2.entity.ResponseEntityBuilder;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.qe.ConnectContext;
import com.google.gson.Gson;
import com.google.gson.reflect.TypeToken;
+import org.apache.commons.lang3.StringUtils;
import org.springfra
Re: [PR] branch-2.1: [improvement](http) improve table schema api for catalog table #45933 [doris]
yiguolei merged PR #45982: URL: https://github.com/apache/doris/pull/45982 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-2.1: [fix](external catalog) Fixed non-master FE log replay during refresh [doris]
zy-kkk opened a new pull request, #46373: URL: https://github.com/apache/doris/pull/46373 cherry-pick #46369 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [fix](external catalog) Fixed non-master FE log replay during refresh [doris]
hello-stephen commented on PR #46373: URL: https://github.com/apache/doris/pull/46373#issuecomment-2568861892 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-2.1: [fix](external catalog) Fixed non-master FE log replay during refresh [doris]
zy-kkk commented on PR #46373: URL: https://github.com/apache/doris/pull/46373#issuecomment-2568861939 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-45933-branch-2.1 deleted (was 2378fb141f2)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-45933-branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git was 2378fb141f2 [improvement](http) improve table schema api for catalog table (#45933) The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cancel) Fix `BufferControlBlock` cancel msg [doris]
github-actions[bot] commented on PR #46266: URL: https://github.com/apache/doris/pull/46266#issuecomment-2568860801 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: branch-3.0: [Bug](scan) do not release tablet_reader on NewOlapScanner::close #46296 (#46354)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new f5ed36065b2 branch-3.0: [Bug](scan) do not release tablet_reader on
NewOlapScanner::close #46296 (#46354)
f5ed36065b2 is described below
commit f5ed36065b24c123b05f01aef0f36b8476bfad96
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Jan 3 16:31:41 2025 +0800
branch-3.0: [Bug](scan) do not release tablet_reader on
NewOlapScanner::close #46296 (#46354)
Cherry-picked from #46296
Co-authored-by: Pxl
---
be/src/vec/exec/scan/new_olap_scanner.cpp | 8
1 file changed, 8 deletions(-)
diff --git a/be/src/vec/exec/scan/new_olap_scanner.cpp
b/be/src/vec/exec/scan/new_olap_scanner.cpp
index f40f30f5b16..f2ca2f0731b 100644
--- a/be/src/vec/exec/scan/new_olap_scanner.cpp
+++ b/be/src/vec/exec/scan/new_olap_scanner.cpp
@@ -524,14 +524,6 @@ Status NewOlapScanner::close(RuntimeState* state) {
return Status::OK();
}
-// olap scan node will call scanner.close() when finished
-// will release resources here
-// if not clear rowset readers in read_params here
-// readers will be release when runtime state deconstructed but
-// deconstructor in reader references runtime state
-// so that it will core
-_tablet_reader_params.rs_splits.clear();
-_tablet_reader.reset();
RETURN_IF_ERROR(VScanner::close(state));
return Status::OK();
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](metaCache)fix bug that names cache can not invalidate. #46287 [doris]
hello-stephen commented on PR #46371: URL: https://github.com/apache/doris/pull/46371#issuecomment-2568848675 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Bug](scan) do not release tablet_reader on NewOlapScanner::close #46296 [doris]
yiguolei merged PR #46354: URL: https://github.com/apache/doris/pull/46354 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-46296-branch-3.0 deleted (was 6cf8a556ac3)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-46296-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git was 6cf8a556ac3 [Bug](scan) do not release tablet_reader on NewOlapScanner::close (#46296) The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](external catalog) Fixed non-master FE log replay during refresh [doris]
zy-kkk commented on PR #46369: URL: https://github.com/apache/doris/pull/46369#issuecomment-2568859978 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/01: [Bug](pipeline) make sink operator process eos signals after wake_up_early (#45207)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch cp_0103_2
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 39697923fa72ac2f0ccf44914ee8e91500611c81
Author: Pxl
AuthorDate: Tue Dec 17 10:35:58 2024 +0800
[Bug](pipeline) make sink operator process eos signals after wake_up_early
(#45207)
1. make sink operator process eos signals after wake_up_early
2. set wake_up_early when `pipeline task meet
wake_up_by_downstream`/`sink reach limit`/`sink get eof status`
3. close non-sink operators after sink meet eos
---
be/src/exprs/runtime_filter.cpp | 6 +-
be/src/pipeline/exec/hashjoin_build_sink.cpp | 57 ++-
be/src/pipeline/exec/hashjoin_probe_operator.cpp | 2 +-
be/src/pipeline/pipeline.cpp | 7 +-
be/src/pipeline/pipeline_task.cpp| 116 +++
be/src/pipeline/pipeline_task.h | 10 +-
6 files changed, 108 insertions(+), 90 deletions(-)
diff --git a/be/src/exprs/runtime_filter.cpp b/be/src/exprs/runtime_filter.cpp
index f5e9c96184f..2169ec727b2 100644
--- a/be/src/exprs/runtime_filter.cpp
+++ b/be/src/exprs/runtime_filter.cpp
@@ -498,12 +498,12 @@ public:
switch (_filter_type) {
case RuntimeFilterType::IN_FILTER: {
if (!_context->hybrid_set) {
-_context->ignored = true;
+set_ignored();
return Status::OK();
}
_context->hybrid_set->insert(wrapper->_context->hybrid_set.get());
if (_max_in_num >= 0 && _context->hybrid_set->size() >=
_max_in_num) {
-_context->ignored = true;
+set_ignored();
// release in filter
_context->hybrid_set.reset();
}
@@ -1332,7 +1332,7 @@ void IRuntimeFilter::set_synced_size(uint64_t
global_size) {
}
void IRuntimeFilter::set_ignored() {
-_wrapper->_context->ignored = true;
+_wrapper->set_ignored();
}
bool IRuntimeFilter::get_ignored() {
diff --git a/be/src/pipeline/exec/hashjoin_build_sink.cpp
b/be/src/pipeline/exec/hashjoin_build_sink.cpp
index 991127d83fe..4fa9f9a95a6 100644
--- a/be/src/pipeline/exec/hashjoin_build_sink.cpp
+++ b/be/src/pipeline/exec/hashjoin_build_sink.cpp
@@ -123,35 +123,48 @@ Status HashJoinBuildSinkLocalState::close(RuntimeState*
state, Status exec_statu
}
}};
-if (!_runtime_filter_slots || _runtime_filters.empty() ||
state->is_cancelled()) {
+if (!_runtime_filter_slots || _runtime_filters.empty() ||
state->is_cancelled() || !_eos) {
return Base::close(state, exec_status);
}
-if (state->get_task()->wake_up_by_downstream()) {
-if (_should_build_hash_table) {
-// partitial ignore rf to make global rf work
+try {
+if (state->get_task()->wake_up_early()) {
+// partitial ignore rf to make global rf work or ignore useless rf
RETURN_IF_ERROR(_runtime_filter_slots->send_filter_size(state, 0,
_finish_dependency));
RETURN_IF_ERROR(_runtime_filter_slots->ignore_all_filters());
-} else {
-// do not publish filter coz local rf not inited and useless
-return Base::close(state, exec_status);
-}
-} else if (_should_build_hash_table) {
-auto* block = _shared_state->build_block.get();
-uint64_t hash_table_size = block ? block->rows() : 0;
-{
-SCOPED_TIMER(_runtime_filter_init_timer);
-RETURN_IF_ERROR(_runtime_filter_slots->init_filters(state,
hash_table_size));
-RETURN_IF_ERROR(_runtime_filter_slots->ignore_filters(state));
-}
-if (hash_table_size > 1) {
-SCOPED_TIMER(_runtime_filter_compute_timer);
-_runtime_filter_slots->insert(block);
+} else if (_should_build_hash_table) {
+auto* block = _shared_state->build_block.get();
+uint64_t hash_table_size = block ? block->rows() : 0;
+{
+SCOPED_TIMER(_runtime_filter_init_timer);
+RETURN_IF_ERROR(_runtime_filter_slots->init_filters(state,
hash_table_size));
+RETURN_IF_ERROR(_runtime_filter_slots->ignore_filters(state));
+}
+if (hash_table_size > 1) {
+SCOPED_TIMER(_runtime_filter_compute_timer);
+_runtime_filter_slots->insert(block);
+}
}
+
+SCOPED_TIMER(_publish_runtime_filter_timer);
+RETURN_IF_ERROR(_runtime_filter_slots->publish(state,
!_should_build_hash_table));
+} catch (Exception& e) {
+bool blocked_by_complete_build_stage = p._shared_hashtable_controller
&&
+
!p._shared_hash_table_context->complete_build_stage;
+bool blocked_by_shared_hash_table_signal = !_should_build_hash_table &&
+
[PR] [Bug](pipeline) make sink operator process eos signals after wake_up_… (#45207) [doris]
BiteThet opened a new pull request, #46374: URL: https://github.com/apache/doris/pull/46374 …early (#45207) 1. make sink operator process eos signals after wake_up_early 2. set wake_up_early when `pipeline task meet wake_up_by_downstream`/`sink reach limit`/`sink get eof status` 3. close non-sink operators after sink meet eos ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](pipeline) make sink operator process eos signals after wake_up_… (#45207) [doris]
hello-stephen commented on PR #46374: URL: https://github.com/apache/doris/pull/46374#issuecomment-2568864084 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](pipeline) make sink operator process eos signals after wake_up_… (#45207) [doris]
BiteThet commented on PR #46374: URL: https://github.com/apache/doris/pull/46374#issuecomment-2568864281 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) parse dict page in ColumnChunkReader::init [doris]
suxiaogang223 commented on PR #46372: URL: https://github.com/apache/doris/pull/46372#issuecomment-2568864461 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch cp_0103_2 created (now 39697923fa7)
This is an automated email from the ASF dual-hosted git repository. panxiaolei pushed a change to branch cp_0103_2 in repository https://gitbox.apache.org/repos/asf/doris.git at 39697923fa7 [Bug](pipeline) make sink operator process eos signals after wake_up_early (#45207) This branch includes the following new commits: new 39697923fa7 [Bug](pipeline) make sink operator process eos signals after wake_up_early (#45207) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: [Improvement](fragment) Use partitioned hash map to manage contexts (… (#46282)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new b2caa40ed7b [Improvement](fragment) Use partitioned hash map to manage
contexts (… (#46282)
b2caa40ed7b is described below
commit b2caa40ed7bd787a35a1a60b1d705eabfea6ce28
Author: Gabriel
AuthorDate: Fri Jan 3 16:46:02 2025 +0800
[Improvement](fragment) Use partitioned hash map to manage contexts (…
(#46282)
…#46235)
Contexts in `fragment_mgr` are managed by a global map and accessed by
multiple threads concurrently with a global lock. It introduced a
obvious overhead. To solve it , this PR use a partitioned hash table to
optimize the global lock.
---
be/src/common/config.cpp| 1 +
be/src/common/config.h | 2 +
be/src/runtime/fragment_mgr.cpp | 508 +++-
be/src/runtime/fragment_mgr.h | 60 +++--
4 files changed, 342 insertions(+), 229 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 587862beffd..342638786ee 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -211,6 +211,7 @@ DEFINE_Int32(check_consistency_worker_count, "1");
DEFINE_Int32(upload_worker_count, "1");
// the count of thread to download
DEFINE_Int32(download_worker_count, "1");
+DEFINE_Int32(num_query_ctx_map_partitions, "128");
// the count of thread to make snapshot
DEFINE_Int32(make_snapshot_worker_count, "5");
// the count of thread to release snapshot
diff --git a/be/src/common/config.h b/be/src/common/config.h
index bcad3ee29a2..5706a38b4b0 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -1350,6 +1350,8 @@ DECLARE_Int32(spill_io_thread_pool_queue_size);
DECLARE_mBool(check_segment_when_build_rowset_meta);
+DECLARE_Int32(num_query_ctx_map_partitions);
+
DECLARE_mBool(enable_s3_rate_limiter);
DECLARE_mInt64(s3_get_bucket_tokens);
DECLARE_mInt64(s3_get_token_per_second);
diff --git a/be/src/runtime/fragment_mgr.cpp b/be/src/runtime/fragment_mgr.cpp
index 87c165222fc..9abdc18ee35 100644
--- a/be/src/runtime/fragment_mgr.cpp
+++ b/be/src/runtime/fragment_mgr.cpp
@@ -225,6 +225,89 @@ static std::map>
_get_all_running_queries
return result;
}
+inline uint32_t get_map_id(const TUniqueId& query_id, size_t capacity) {
+uint32_t value = HashUtil::hash(&query_id.lo, 8, 0);
+value = HashUtil::hash(&query_id.hi, 8, value);
+return value % capacity;
+}
+
+inline uint32_t get_map_id(std::pair key, size_t capacity) {
+uint32_t value = HashUtil::hash(&key.first.lo, 8, 0);
+value = HashUtil::hash(&key.first.hi, 8, value);
+return value % capacity;
+}
+
+template
+ConcurrentContextMap::ConcurrentContextMap() {
+_internal_map.resize(config::num_query_ctx_map_partitions);
+for (size_t i = 0; i < config::num_query_ctx_map_partitions; i++) {
+_internal_map[i] = {std::make_unique(),
+phmap::flat_hash_map()};
+}
+}
+
+template
+Value ConcurrentContextMap::find(const Key& query_id) {
+auto id = get_map_id(query_id, _internal_map.size());
+{
+std::shared_lock lock(*_internal_map[id].first);
+auto& map = _internal_map[id].second;
+auto search = map.find(query_id);
+if (search != map.end()) {
+return search->second;
+}
+return std::shared_ptr(nullptr);
+}
+}
+
+template
+Status ConcurrentContextMap::apply_if_not_exists(
+const Key& query_id, std::shared_ptr& query_ctx,
ApplyFunction&& function) {
+auto id = get_map_id(query_id, _internal_map.size());
+{
+std::unique_lock lock(*_internal_map[id].first);
+auto& map = _internal_map[id].second;
+auto search = map.find(query_id);
+if (search != map.end()) {
+query_ctx = search->second.lock();
+}
+if (!query_ctx) {
+return function(map);
+}
+return Status::OK();
+}
+}
+
+template
+void ConcurrentContextMap::erase(const Key& query_id) {
+auto id = get_map_id(query_id, _internal_map.size());
+{
+std::unique_lock lock(*_internal_map[id].first);
+auto& map = _internal_map[id].second;
+map.erase(query_id);
+}
+}
+
+template
+void ConcurrentContextMap::insert(const Key& query_id,
+
std::shared_ptr query_ctx) {
+auto id = get_map_id(query_id, _internal_map.size());
+{
+std::unique_lock lock(*_internal_map[id].first);
+auto& map = _internal_map[id].second;
+map.insert({query_id, query_ctx});
+}
+}
+
+template
+void ConcurrentContextMap::clear() {
+for (auto& pair : _internal_map) {
+std::unique_lock lock(*pair.first);
+auto& map = pair.second;
+map.clear();
+}
+}
+
Fr
Re: [PR] [Improvement](fragment) Use partitioned hash map to manage contexts (… [doris]
Gabriel39 merged PR #46282: URL: https://github.com/apache/doris/pull/46282 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) set correct sort key for aggregate #45369 branch-3.0 [doris]
doris-robot commented on PR #45706: URL: https://github.com/apache/doris/pull/45706#issuecomment-2569970753 ClickBench: Total hot run time: 33.38 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 4ca195e9a744e5973c0c26d3063a14ebf726c504, data reload: false query1 0.040.030.03 query2 0.070.030.03 query3 0.230.070.07 query4 1.630.100.10 query5 0.530.500.51 query6 1.140.720.72 query7 0.020.020.02 query8 0.040.040.05 query9 0.570.520.51 query10 0.570.570.57 query11 0.150.110.10 query12 0.140.120.11 query13 0.610.600.60 query14 2.963.032.93 query15 0.890.830.83 query16 0.390.370.40 query17 1.081.021.06 query18 0.230.220.21 query19 1.951.771.96 query20 0.020.010.01 query21 15.35 0.600.57 query22 2.492.001.76 query23 17.24 0.950.78 query24 2.641.261.35 query25 0.270.200.14 query26 0.240.130.13 query27 0.040.050.04 query28 10.54 1.091.07 query29 12.62 3.303.29 query30 0.250.070.06 query31 2.840.400.39 query32 3.250.460.46 query33 2.983.023.05 query34 17.10 4.534.54 query35 4.544.564.64 query36 0.670.490.48 query37 0.100.060.06 query38 0.050.030.04 query39 0.040.030.02 query40 0.160.120.12 query41 0.090.030.02 query42 0.040.020.02 query43 0.040.030.03 Total cold run time: 106.84 s Total hot run time: 33.38 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) set correct sort key for aggregate #45369 branch-3.0 [doris]
doris-robot commented on PR #45706: URL: https://github.com/apache/doris/pull/45706#issuecomment-2569968598 TPC-DS: Total hot run time: 191812 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 4ca195e9a744e5973c0c26d3063a14ebf726c504, data reload: false query1 976 380 360 360 query2 6514209720062006 query3 6704214 222 214 query4 34212 23497 23407 23407 query5 4395469 466 466 query6 296 186 178 178 query7 4625315 315 315 query8 319 230 230 230 query9 9720266326702663 query10 469 266 282 266 query11 18158 15273 15239 15239 query12 153 106 101 101 query13 1643432 427 427 query14 8938762567726772 query15 251 173 183 173 query16 8146447 464 447 query17 1637563 548 548 query18 2148303 303 303 query19 378 155 148 148 query20 122 105 109 105 query21 60 45 44 44 query22 4516419842014198 query23 35797 34941 33959 33959 query24 11768 287228452845 query25 544 377 383 377 query26 1102175 172 172 query27 2800352 352 352 query28 7935241124012401 query29 662 434 421 421 query30 353 168 165 165 query31 1022798 801 798 query32 93 56 61 56 query33 784 281 285 281 query34 958 499 507 499 query35 915 714 718 714 query36 1125935 993 935 query37 213 81 79 79 query38 4192402639733973 query39 1494143514241424 query40 235 85 88 85 query41 54 49 50 49 query42 117 105 103 103 query43 537 498 481 481 query44 1288811 811 811 query45 189 165 168 165 query46 1163718 723 718 query47 1917182418261824 query48 492 387 384 384 query49 1294399 397 397 query50 820 416 416 416 query51 7244713671047104 query52 105 95 93 93 query53 267 190 190 190 query54 1079471 474 471 query55 86 83 85 83 query56 294 254 257 254 query57 1227112811321128 query58 250 215 224 215 query59 3218308128662866 query60 303 267 273 267 query61 135 130 148 130 query62 887 683 681 681 query63 231 197 194 194 query64 5282774 735 735 query65 3313325332313231 query66 1297333 333 333 query67 15986 15645 15874 15645 query68 4932591 572 572 query69 470 284 274 274 query70 1186110611631106 query71 350 276 279 276 query72 6226394039383938 query73 787 354 349 349 query74 10509 907189548954 query75 3384267026762670 query76 2913114210841084 query77 468 290 267 267 query78 10524 961195629562 query79 1130595 602 595 query80 799 452 436 436 query81 534 243 236 236 query82 1313124 119 119 query83 235 146 148 146 query84 246 84 84 84 query85 1117308 309 308 query86 313 308 291 291 query87 4481457443314331 query88 3577243423712371 query89 414 291 306 291 query90 2117194 194 194 query91 186 147 149 147 query92 63 51 51 51 query93 1047559 550 550 query94 796 298 284 284 query95 358 262 260 260 query96 625 281 292 281 query97 3325323131773177 query98 219 208 203 203 query99 1723134612911291 Total cold run time: 301814 ms Total hot run time: 191812 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
(doris-website) branch asf-site updated (96a5cb518f8 -> 6caed5cea1d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard 96a5cb518f8 Automated deployment with doris branch @ 8512da5c344b974360b07a303308f7edf1b5350e new 6caed5cea1d Automated deployment with doris branch @ 8512da5c344b974360b07a303308f7edf1b5350e This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (96a5cb518f8) \ N -- N -- N refs/heads/asf-site (6caed5cea1d) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/1.2/search-index.json | 2 +- docs/2.0/search-index.json | 2 +- docs/3.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/1.2/search-index.json | 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/3.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 10 files changed, 10 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](variant) fix schema change for variant from not null to null (#46403)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new db224ba15ff [fix](variant) fix schema change for variant from not null
to null (#46403)
db224ba15ff is described below
commit db224ba15ff2ed4c10f9835ce7ba1d026c1f45d4
Author: lihangyu
AuthorDate: Sat Jan 4 09:00:43 2025 +0800
[fix](variant) fix schema change for variant from not null to null (#46403)
cherry-pick from #46279
---
be/src/vec/common/schema_util.cpp | 29 +++--
.../variant_p0/schema_change/schema_change.out | 10
.../variant_p0/schema_change/schema_change.groovy | 30 ++
3 files changed, 61 insertions(+), 8 deletions(-)
diff --git a/be/src/vec/common/schema_util.cpp
b/be/src/vec/common/schema_util.cpp
index 51a3ed8c317..e1df98a0d72 100644
--- a/be/src/vec/common/schema_util.cpp
+++ b/be/src/vec/common/schema_util.cpp
@@ -148,20 +148,17 @@ bool is_conversion_required_between_integers(const
TypeIndex& lhs, const TypeInd
Status cast_column(const ColumnWithTypeAndName& arg, const DataTypePtr& type,
ColumnPtr* result) {
ColumnsWithTypeAndName arguments {arg, {nullptr, type, type->get_name()}};
-auto function = SimpleFunctionFactory::instance().get_function("CAST",
arguments, type);
-if (!function) {
-return Status::InternalError("Not found cast function {} to {}",
arg.type->get_name(),
- type->get_name());
-}
-Block tmp_block {arguments};
-size_t result_column = tmp_block.columns();
-auto ctx = FunctionContext::create_context(nullptr, {}, {});
// To prevent from null info lost, we should not call function since the
function framework will wrap
// nullable to Variant instead of the root of Variant
// correct output: Nullable(Array(int)) ->
Nullable(Variant(Nullable(Array(int
// incorrect output: Nullable(Array(int)) -> Nullable(Variant(Array(int)))
if (WhichDataType(remove_nullable(type)).is_variant_type()) {
+// If source column is variant, so the nullable info is different from
dst column
+if (WhichDataType(remove_nullable(arg.type)).is_variant_type()) {
+*result = type->is_nullable() ? make_nullable(arg.column) :
remove_nullable(arg.column);
+return Status::OK();
+}
// set variant root column/type to from column/type
auto variant = ColumnObject::create(true /*always nullable*/);
CHECK(arg.column->is_nullable());
@@ -173,6 +170,22 @@ Status cast_column(const ColumnWithTypeAndName& arg, const
DataTypePtr& type, Co
return Status::OK();
}
+auto function = SimpleFunctionFactory::instance().get_function("CAST",
arguments, type);
+if (!function) {
+return Status::InternalError("Not found cast function {} to {}",
arg.type->get_name(),
+ type->get_name());
+}
+Block tmp_block {arguments};
+size_t result_column = tmp_block.columns();
+auto ctx = FunctionContext::create_context(nullptr, {}, {});
+
+if (WhichDataType(arg.type).is_nothing()) {
+// cast from nothing to any type should result in nulls
+*result =
type->create_column_const_with_default_value(arg.column->size())
+ ->convert_to_full_column_if_const();
+return Status::OK();
+}
+
// We convert column string to jsonb type just add a string jsonb field to
dst column instead of parse
// each line in original string column.
ctx->set_string_as_jsonb_string(true);
diff --git a/regression-test/data/variant_p0/schema_change/schema_change.out
b/regression-test/data/variant_p0/schema_change/schema_change.out
index 4956e8311b0..cbc4488d93b 100644
--- a/regression-test/data/variant_p0/schema_change/schema_change.out
+++ b/regression-test/data/variant_p0/schema_change/schema_change.out
@@ -71,3 +71,13 @@
1 hello world
1 hello world
+-- !sql --
+1 {"a":1.0}
+2 {"a":111.}
+3 {"a":"1"}
+
+-- !sql --
+1 {"a":1.0}
+2 {"a":111.}
+3 {"a":"1"}
+
diff --git
a/regression-test/suites/variant_p0/schema_change/schema_change.groovy
b/regression-test/suites/variant_p0/schema_change/schema_change.groovy
index 877874b7e03..9c39b3ad8ba 100644
--- a/regression-test/suites/variant_p0/schema_change/schema_change.groovy
+++ b/regression-test/suites/variant_p0/schema_change/schema_change.groovy
@@ -79,4 +79,34 @@ suite("regression_test_variant_schema_change",
"variant_type"){
sql """INSERT INTO ${table_name} SELECT k, v,v from ${table_name} limit
"""
// select from mv
qt_sql """select v['k1'], cast(v['k2'] as string) from ${table_name} order
by k desc limit 10"""
+
+// not null to null
+sql "drop table if exis
Re: [PR] branch-2.1: [fix](regression)fix hive translation unstable case. #46385 [doris]
yiguolei merged PR #46409: URL: https://github.com/apache/doris/pull/46409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: branch-2.1: [fix](regression)fix hive translation unstable case. #46385 (#46409)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new d8c94d6392c branch-2.1: [fix](regression)fix hive translation unstable
case. #46385 (#46409)
d8c94d6392c is described below
commit d8c94d6392ccadaa514df4d03b4e6e9981871ea8
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Sat Jan 4 08:59:56 2025 +0800
branch-2.1: [fix](regression)fix hive translation unstable case. #46385
(#46409)
Cherry-picked from #46385
Co-authored-by: daidai
---
.../hive/scripts/create_preinstalled_scripts/run25.hql | 7 ---
.../hive/test_hive_translation_insert_only.groovy | 1 +
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git
a/docker/thirdparties/docker-compose/hive/scripts/create_preinstalled_scripts/run25.hql
b/docker/thirdparties/docker-compose/hive/scripts/create_preinstalled_scripts/run25.hql
index da6400bdff0..66e73f51df8 100755
---
a/docker/thirdparties/docker-compose/hive/scripts/create_preinstalled_scripts/run25.hql
+++
b/docker/thirdparties/docker-compose/hive/scripts/create_preinstalled_scripts/run25.hql
@@ -61,7 +61,8 @@ STORED AS ORC;
INSERT INTO TABLE orc_to_acid_compacted_tb PARTITION (part_col=101) VALUES (1,
'A'), (3, 'C');
INSERT INTO TABLE orc_to_acid_compacted_tb PARTITION (part_col=102) VALUES (2,
'B');
ALTER TABLE orc_to_acid_compacted_tb SET TBLPROPERTIES
('transactional'='true');
-ALTER TABLE orc_to_acid_compacted_tb COMPACT 'major';
+ALTER TABLE orc_to_acid_compacted_tb partition(part_col='101') COMPACT 'major'
and wait;
+ALTER TABLE orc_to_acid_compacted_tb partition(part_col='102') COMPACT 'major'
and wait;
INSERT INTO TABLE orc_to_acid_compacted_tb PARTITION (part_col=102) VALUES (4,
'D');
update orc_to_acid_compacted_tb set value = "CC" where id = 3;
update orc_to_acid_compacted_tb set value = "BB" where id = 2;
@@ -75,7 +76,7 @@ insert into orc_acid_minor values (1, 'A');
insert into orc_acid_minor values (2, 'B');
insert into orc_acid_minor values (3, 'C');
update orc_acid_minor set value = "BB" where id = 2;
-ALTER TABLE orc_acid_minor COMPACT 'minor';
+ALTER TABLE orc_acid_minor COMPACT 'minor' and wait;
insert into orc_acid_minor values (4, 'D');
update orc_acid_minor set value = "DD" where id = 4;
DELETE FROM orc_acid_minor WHERE id = 3;
@@ -89,7 +90,7 @@ insert into orc_acid_major values (1, 'A');
insert into orc_acid_major values (2, 'B');
insert into orc_acid_major values (3, 'C');
update orc_acid_major set value = "BB" where id = 2;
-ALTER TABLE orc_acid_major COMPACT 'minor';
+ALTER TABLE orc_acid_major COMPACT 'minor' and wait;
insert into orc_acid_major values (4, 'D');
update orc_acid_major set value = "DD" where id = 4;
DELETE FROM orc_acid_major WHERE id = 3;
diff --git
a/regression-test/suites/external_table_p2/hive/test_hive_translation_insert_only.groovy
b/regression-test/suites/external_table_p2/hive/test_hive_translation_insert_only.groovy
index e9342161d49..9b021e1dc81 100644
---
a/regression-test/suites/external_table_p2/hive/test_hive_translation_insert_only.groovy
+++
b/regression-test/suites/external_table_p2/hive/test_hive_translation_insert_only.groovy
@@ -21,6 +21,7 @@ suite("test_hive_translation_insert_only",
"p2,external,hive,external_remote,ext
//hudi hive use same catalog in p2.
if (enabled == null || !enabled.equalsIgnoreCase("true")) {
logger.info("disable test")
+return;
}
String props = context.config.otherConfigs.get("hudiEmrCatalog")
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
