[GitHub] [doris] yiguolei merged pull request #16117: [Fix](Oracle External Table) fix that oracle external table can not insert batch values
yiguolei merged PR #16117: URL: https://github.com/apache/doris/pull/16117 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16119: [test](pipline)Remove P1 regression required check in .asf.yaml
yiguolei merged PR #16119: URL: https://github.com/apache/doris/pull/16119 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16088: [fix](statistics) Fix statistics related threads continuously spawn as doing checkpoint
yiguolei merged PR #16088: URL: https://github.com/apache/doris/pull/16088 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16118: [debug](ParquetReader) print file path in error message if read parquet failed
yiguolei merged PR #16118: URL: https://github.com/apache/doris/pull/16118 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16051: [regression-test](sub query) add regression test for subquery with limit
yiguolei merged PR #16051: URL: https://github.com/apache/doris/pull/16051 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16050: [refactor](javaudf) simplify UdfExecutor and UdafExecutor
yiguolei merged PR #16050: URL: https://github.com/apache/doris/pull/16050 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #15943: [Feature](ES): catalog support mapping es _id
yiguolei merged PR #15943: URL: https://github.com/apache/doris/pull/15943 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #15875: [Feature](profile) set sql and defaultDb fields in show-load-profile.
yiguolei merged PR #15875: URL: https://github.com/apache/doris/pull/15875 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #15843: [vectorzied](jdbc) fix jdbc executor for get result by batch and memo…
yiguolei merged PR #15843: URL: https://github.com/apache/doris/pull/15843 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #15769: [Fix](storage) Fix bug that cooldown time for partition in SSD is wrong
yiguolei merged PR #15769: URL: https://github.com/apache/doris/pull/15769 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #16102: [Bug](runtimefilter) Fix wrong runtime filter on datetime
englefly commented on code in PR #16102:
URL: https://github.com/apache/doris/pull/16102#discussion_r1083228216
##
be/src/exprs/bloomfilter_predicate.h:
##
@@ -334,6 +334,12 @@ struct DateTimeFindOp : public CommonFindOp
{
value.from_olap_datetime(*reinterpret_cast(data));
return bloom_filter.test(Slice((char*)&value, sizeof(DateTimeValue)));
}
+void insert(BloomFilterAdaptor& bloom_filter, const void* data) const {
Review Comment:
who invokes this `insert`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #16102: [Bug](runtimefilter) Fix wrong runtime filter on datetime
englefly commented on code in PR #16102:
URL: https://github.com/apache/doris/pull/16102#discussion_r1083228825
##
be/src/exprs/bloomfilter_predicate.h:
##
@@ -334,6 +334,12 @@ struct DateTimeFindOp : public CommonFindOp
{
value.from_olap_datetime(*reinterpret_cast(data));
return bloom_filter.test(Slice((char*)&value, sizeof(DateTimeValue)));
}
+void insert(BloomFilterAdaptor& bloom_filter, const void* data) const {
Review Comment:
shall we do the unify job of all datatypes in one place? There are other
data types whose storage format are different from their engine format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16121: [refactor](remoe non vec code) remove non vectorized conjunctx from scanner
yiguolei merged PR #16121: URL: https://github.com/apache/doris/pull/16121 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei opened a new pull request, #16128: [refactor](remove broker scan node) it is not useful any more
yiguolei opened a new pull request, #16128: URL: https://github.com/apache/doris/pull/16128 1. remove broker scannode 2. remove broker table 3. remove broker scanner 4. remove json scanner 5. remove orc scanner 6. remove hive external table 7. remove hudi external table # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 8. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 9. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 10. Does it need to update dependencies: - [ ] Yes - [ ] No 11. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16128: [refactor](remove broker scan node) it is not useful any more
hello-stephen commented on PR #16128: URL: https://github.com/apache/doris/pull/16128#issuecomment-1399247573 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.5 seconds load time: 493 seconds storage size: 17122617434 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230121130129_clickbench_pr_84865.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #16129: [cleanup](Nereids): cleanup useless code.
jackwener opened a new pull request, #16129: URL: https://github.com/apache/doris/pull/16129 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16129: [cleanup](Nereids): cleanup useless code.
hello-stephen commented on PR #16129: URL: https://github.com/apache/doris/pull/16129#issuecomment-1399257309 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.17 seconds load time: 481 seconds storage size: 17122137225 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230121140228_clickbench_pr_84875.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16127: [deps](libhdfs3) update to 2.3.6 to fix kms aes 256 bug
yiguolei merged PR #16127: URL: https://github.com/apache/doris/pull/16127 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on a diff in pull request #15966: [Feature](map)support complex struct for doris
amorynan commented on code in PR #15966:
URL: https://github.com/apache/doris/pull/15966#discussion_r1082024425
##
be/src/olap/rowset/segment_v2/column_writer.cpp:
##
@@ -173,6 +173,119 @@ Status ColumnWriter::create(const ColumnWriterOptions&
opts, const TabletColumn*
*writer = std::move(writer_local);
return Status::OK();
}
+ case FieldType::OLAP_FIELD_TYPE_MAP: {
+DCHECK(column->get_subtype_count() == 2);
+// todo . here key and value is array only?
+const TabletColumn& key_column = column->get_sub_column(0); //
field_type is true key and value
+const TabletColumn& value_column = column->get_sub_column(1);
+
+// create null writer
+ScalarColumnWriter* null_writer = nullptr;
+if (opts.meta->is_nullable()) {
+FieldType null_type = FieldType::OLAP_FIELD_TYPE_TINYINT;
+ColumnWriterOptions null_options;
+null_options.meta = opts.meta->add_children_columns();
+null_options.meta->set_column_id(3);
+null_options.meta->set_unique_id(3);
+null_options.meta->set_type(null_type);
+null_options.meta->set_is_nullable(false);
+null_options.meta->set_length(
+
get_scalar_type_info()->size());
+null_options.meta->set_encoding(DEFAULT_ENCODING);
+null_options.meta->set_compression(opts.meta->compression());
+
+null_options.need_zone_map = false;
+null_options.need_bloom_filter = false;
+null_options.need_bitmap_index = false;
+
+TabletColumn null_column = TabletColumn(
+OLAP_FIELD_AGGREGATION_NONE, null_type, false,
+null_options.meta->unique_id(),
null_options.meta->length());
+null_column.set_name("nullable");
+null_column.set_index_length(-1); // no short key index
+std::unique_ptr
null_field(FieldFactory::create(null_column));
+null_writer =
+new ScalarColumnWriter(null_options,
std::move(null_field), file_writer);
+}
+
+// create key writer
+std::unique_ptr key_writer;
+ ColumnWriterOptions key_opts;
+TabletColumn key_list_column(OLAP_FIELD_AGGREGATION_NONE,
OLAP_FIELD_TYPE_ARRAY);
+{
+
key_list_column.add_sub_column(const_cast(key_column));
+//key_list_column.add_sub_column(key_column);
+key_list_column.set_name("map.key");
+key_list_column.set_index_length(-1);
+
+
+key_opts.meta = opts.meta->mutable_children_columns(0);
Review Comment:
is same as null writer ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on a diff in pull request #15966: [Feature](map)support complex struct for doris
amorynan commented on code in PR #15966:
URL: https://github.com/apache/doris/pull/15966#discussion_r1083389580
##
be/src/olap/rowset/segment_v2/column_writer.cpp:
##
@@ -173,6 +173,119 @@ Status ColumnWriter::create(const ColumnWriterOptions&
opts, const TabletColumn*
*writer = std::move(writer_local);
return Status::OK();
}
+ case FieldType::OLAP_FIELD_TYPE_MAP: {
+DCHECK(column->get_subtype_count() == 2);
+// todo . here key and value is array only?
+const TabletColumn& key_column = column->get_sub_column(0); //
field_type is true key and value
+const TabletColumn& value_column = column->get_sub_column(1);
+
+// create null writer
+ScalarColumnWriter* null_writer = nullptr;
+if (opts.meta->is_nullable()) {
+FieldType null_type = FieldType::OLAP_FIELD_TYPE_TINYINT;
+ColumnWriterOptions null_options;
+null_options.meta = opts.meta->add_children_columns();
+null_options.meta->set_column_id(3);
+null_options.meta->set_unique_id(3);
+null_options.meta->set_type(null_type);
+null_options.meta->set_is_nullable(false);
+null_options.meta->set_length(
+
get_scalar_type_info()->size());
+null_options.meta->set_encoding(DEFAULT_ENCODING);
+null_options.meta->set_compression(opts.meta->compression());
+
+null_options.need_zone_map = false;
+null_options.need_bloom_filter = false;
+null_options.need_bitmap_index = false;
+
+TabletColumn null_column = TabletColumn(
+OLAP_FIELD_AGGREGATION_NONE, null_type, false,
+null_options.meta->unique_id(),
null_options.meta->length());
+null_column.set_name("nullable");
+null_column.set_index_length(-1); // no short key index
+std::unique_ptr
null_field(FieldFactory::create(null_column));
+null_writer =
+new ScalarColumnWriter(null_options,
std::move(null_field), file_writer);
+}
+
+// create key writer
+std::unique_ptr key_writer;
+ ColumnWriterOptions key_opts;
+TabletColumn key_list_column(OLAP_FIELD_AGGREGATION_NONE,
OLAP_FIELD_TYPE_ARRAY);
+{
+
key_list_column.add_sub_column(const_cast(key_column));
+//key_list_column.add_sub_column(key_column);
+key_list_column.set_name("map.key");
+key_list_column.set_index_length(-1);
+
+
+key_opts.meta = opts.meta->mutable_children_columns(0);
Review Comment:
but I need make new array columns as map key and value column writer because
I need wrap actually typed column with array column which actually may not init
in init_column_meta(), what shoud I deal with the
column_id/unique_id/encoding/compression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 opened a new pull request, #16130: [test](pipeline) Run nereids cases in p1/p2
Kikyou1997 opened a new pull request, #16130: URL: https://github.com/apache/doris/pull/16130 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee merged pull request #15925: [Refactor]Merged string_value into string_ref
HappenLee merged PR #15925: URL: https://github.com/apache/doris/pull/15925 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yukang-Lian opened a new issue, #16131: [Bug] cast to datetimev2 return wrong result
Yukang-Lian opened a new issue, #16131:
URL: https://github.com/apache/doris/issues/16131
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no
similar issues.
### Version
master
### What's Wrong?
mysql> select cast ('2022-12-02 22:23:24.99' as datetimev2(4)),cast
('2022-12-02 22:23:23.99' as datetimev2(4)) from test;
+-+-+
| CAST('2022-12-02 22:23:24.99' AS DATETIMEV2(4)) | CAST('2022-12-02
22:23:23.99' AS DATETIMEV2(4)) |
+-+-+
| 2022-12-02 22:23:24.00 | 2022-12-02
22:23:23.00 |
+-+-+
1 row in set (0.03 sec)
mysql> mysql> select cast ('2022-12-02 22:23:24.99' as
datetimev2(4)),cast ('2022-12-02 22:23:23.99' as datetimev2(4));
+-+-+
| CAST('2022-12-02 22:23:24.99' AS DATETIMEV2(4)) | CAST('2022-12-02
22:23:23.99' AS DATETIMEV2(4)) |
+-+-+
| 2022-12-02 22:23:24.00 | 2022-12-02
22:23:23.00 |
+-+-+
1 row in set (0.00 sec)
### What You Expected?
```sql
mysql> select cast ('2022-12-02 22:23:24.99' as datetimev2(4)),cast
('2022-12-02 22:23:23.99' as datetimev2(4)) from test;
+-+-+
| CAST('2022-12-02 22:23:24.99' AS DATETIMEV2(4)) | CAST('2022-12-02
22:23:23.99' AS DATETIMEV2(4)) |
+-+-+
| 2022-12-02 22:23:24. | 2022-12-02
22:23:23. |
+-+-+
1 row in set (0.03 sec)
mysql> select cast ('2022-12-02 22:23:24.99' as datetimev2(4)),cast
('2022-12-02 22:23:23.99' as datetimev2(4));
+-+-+
| CAST('2022-12-02 22:23:24.99' AS DATETIMEV2(4)) | CAST('2022-12-02
22:23:23.99' AS DATETIMEV2(4)) |
+-+-+
| 2022-12-02 22:23:24. | 2022-12-02
22:23:23. |
+-+-+
1 row in set (0.00 sec)
```
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yukang-Lian opened a new pull request, #16132: [fix](datetimev2) fix cast to datetimev2 return wrong result
Yukang-Lian opened a new pull request, #16132: URL: https://github.com/apache/doris/pull/16132 # Proposed changes Issue Number: close #16131 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16132: [fix](datetimev2) fix cast to datetimev2 return wrong result
hello-stephen commented on PR #16132: URL: https://github.com/apache/doris/pull/16132#issuecomment-1399506271 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.96 seconds load time: 486 seconds storage size: 17122433194 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230122143650_clickbench_pr_84927.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #15173: fix(sec): upgrade org.apache.hive:hive-exec to 3.1.3
morningman commented on PR #15173: URL: https://github.com/apache/doris/pull/15173#issuecomment-1399514382 fixed in #16046 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed pull request #15173: fix(sec): upgrade org.apache.hive:hive-exec to 3.1.3
morningman closed pull request #15173: fix(sec): upgrade org.apache.hive:hive-exec to 3.1.3 URL: https://github.com/apache/doris/pull/15173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #15801: [Enhancement](export) cancel all running coordinators when execute ca…
morningman merged PR #15801: URL: https://github.com/apache/doris/pull/15801 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #15128: [Feature](export) Support cancel export statement
morningman commented on code in PR #15128:
URL: https://github.com/apache/doris/pull/15128#discussion_r1061506780
##
fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java:
##
@@ -294,12 +351,12 @@ public void replayCreateExportJob(ExportJob job) {
}
public void replayUpdateJobState(long jobId, ExportJob.JobState newState) {
-writeLock();
+readLock();
Review Comment:
Why change to read lock?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yukang-Lian closed issue #15936: [Bug] array(datetimev2) return wrong result
Yukang-Lian closed issue #15936: [Bug] array(datetimev2) return wrong result URL: https://github.com/apache/doris/issues/15936 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] tang350476527 commented on pull request #16132: [fix](datetimev2) fix cast to datetimev2 return wrong result
tang350476527 commented on PR #16132: URL: https://github.com/apache/doris/pull/16132#issuecomment-1399524874 谢谢您的来信，您的邮件已收到！ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-thirdparty] morningman merged pull request #33: [doc] update version to 2.3.6 and update changelog
morningman merged PR #33: URL: https://github.com/apache/doris-thirdparty/pull/33 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-thirdparty] morningman merged pull request #34: [doc] update readme
morningman merged PR #34: URL: https://github.com/apache/doris-thirdparty/pull/34 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #16050: [refactor](javaudf) simplify UdfExecutor and UdafExecutor
morningman commented on PR #16050: URL: https://github.com/apache/doris/pull/16050#issuecomment-1399782437 Please add some comment next time, or other reviewer can' t get your point. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16074: [vectorized](analytic) fix analytic node of window function get wrong…
yiguolei merged PR #16074: URL: https://github.com/apache/doris/pull/16074 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #15951: [Bug] 基础函数rank()开窗排序结果错误
yiguolei closed issue #15951: [Bug] 基础函数rank()开窗排序结果错误 URL: https://github.com/apache/doris/issues/15951 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chenlinzhong closed pull request #15544: [mtmv](ut) add mtmv ut
chenlinzhong closed pull request #15544: [mtmv](ut) add mtmv ut URL: https://github.com/apache/doris/pull/15544 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed pull request #16017: Revert "[enhancement](rewrite) Remove unused wide common factors to improve scan performance in ExtractCommonFactorsRule"
morningman closed pull request #16017: Revert "[enhancement](rewrite) Remove unused wide common factors to improve scan performance in ExtractCommonFactorsRule" URL: https://github.com/apache/doris/pull/16017 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16128: [refactor](remove broker scan node) it is not useful any more
yiguolei merged PR #16128: URL: https://github.com/apache/doris/pull/16128 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #16133: [fix](icebergv2) fix bug that delete file reader is not opened
morningman opened a new pull request, #16133: URL: https://github.com/apache/doris/pull/16133 # Proposed changes Issue Number: close #xxx ## Problem summary This pr #15836 change the way to use parquet reader by first `open()` then `init_reader()`. But we forgot to call `open()` for iceberg delete file, which cause coredump. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16133: [fix](icebergv2) fix bug that delete file reader is not opened
hello-stephen commented on PR #16133: URL: https://github.com/apache/doris/pull/16133#issuecomment-1400271675 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.74 seconds load time: 494 seconds storage size: 17122798066 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230123123423_clickbench_pr_85027.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #16135: [fix](icebergv2) fix icebergv2 delete file open bug and column case insensitive
morningman opened a new pull request, #16135: URL: https://github.com/apache/doris/pull/16135 # Proposed changes Issue Number: close #xxx ## Problem summary cherry-pick #16133 and #15156 ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #16134: [regression](external p2) cherry pick external table p2 case from master
morningman merged PR #16134: URL: https://github.com/apache/doris/pull/16134 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #16134: [regression](external p2) cherry pick external table p2 case from master
morningman opened a new pull request, #16134: URL: https://github.com/apache/doris/pull/16134 # Proposed changes Issue Number: close #xxx ## Problem summary Cherry pick p2 case of external table from master ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #15511: [feature](Load)Suppot MySQL Load Data
morningman commented on code in PR #15511:
URL: https://github.com/apache/doris/pull/15511#discussion_r1084074726
##
fe/fe-core/src/main/cup/sql_parser.cup:
##
@@ -618,9 +618,13 @@ terminal String
KW_MTMV,
KW_TYPECAST,
KW_HISTOGRAM,
+<<< HEAD
Review Comment:
conflict
##
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadManager.java:
##
@@ -151,6 +181,183 @@ public void createLoadJobV1FromMultiStart(String
fullDbName, String label) throw
}
}
+public LoadJobRowResult executeMySqlLoadJobFromStmt(ConnectContext
context, LoadStmt stmt)
+throws IOException, LoadException {
+LoadJobRowResult loadResult = new LoadJobRowResult();
Review Comment:
How about moving these methods to a separate class to make LoadManager
simple?
##
fe/fe-common/src/main/java/org/apache/doris/common/io/ByteBufferNetworkInputStream.java:
##
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.io;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+public class ByteBufferNetworkInputStream extends InputStream {
+private ArrayBlockingQueue queue;
+private ByteArrayInputStream currentInputStream;
+private volatile boolean finished = false;
+private volatile boolean closed = false;
+
+public ByteBufferNetworkInputStream() {
+this(32);
+}
+
+public ByteBufferNetworkInputStream(int capacity) {
+this.queue = new ArrayBlockingQueue<>(capacity);
+}
+
+public void fillByteBuffer(ByteBuffer buffer) throws IOException,
InterruptedException {
+if (closed) {
+throw new IOException("Stream is already closed.");
+}
+ByteArrayInputStream inputStream = new
ByteArrayInputStream(buffer.array(), buffer.position(), buffer.limit());
+queue.offer(inputStream, 300, TimeUnit.SECONDS);
Review Comment:
Is there any reason to set timeout to 300s?
##
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadManager.java:
##
@@ -151,6 +181,183 @@ public void createLoadJobV1FromMultiStart(String
fullDbName, String label) throw
}
}
+public LoadJobRowResult executeMySqlLoadJobFromStmt(ConnectContext
context, LoadStmt stmt)
+throws IOException, LoadException {
+LoadJobRowResult loadResult = new LoadJobRowResult();
+// Mysql data load only have one data desc
+DataDescription dataDesc = stmt.getDataDescriptions().get(0);
+String database = dataDesc.getDbName();
+String table = dataDesc.getTableName();
+List filePaths = dataDesc.getFilePaths();
+try (final CloseableHttpClient httpclient =
HttpClients.createDefault()) {
+for (String file : filePaths) {
+InputStreamEntity entity = getInputStreamEntity(context,
dataDesc.isClientLocal(), file);
+HttpPut request = generateRequestForMySqlLoad(entity,
dataDesc, database, table);
+try (final CloseableHttpResponse response =
httpclient.execute(request)) {
+JSONObject result =
JSON.parseObject(EntityUtils.toString(response.getEntity()));
+if
(!result.getString("Status").equalsIgnoreCase("Success")) {
+LOG.warn("Execute stream load for mysql data load
failed with message: " + request);
+throw new LoadException(result.getString("Message"));
+}
+loadResult.incRecords(result.getLong("NumberLoadedRows"));
+
loadResult.incSkipped(result.getIntValue("NumberFilteredRows"));
+}
+}
+}
+return loadResult;
+}
+
+private InputStreamEntity getInputStreamEntity(ConnectContext context,
boolean isClintLocal, String file)
+throws IOException {
+InputStream inputStream;
+if (isClintLocal) {
Review Comment:
```suggestion
if (isClientLocal) {
``
[GitHub] [doris] yiguolei opened a new pull request, #16136: [refactor](remove expr) remove unused Expr and ExprContext related codes
yiguolei opened a new pull request, #16136: URL: https://github.com/apache/doris/pull/16136 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16136: [refactor](remove expr) remove non vectorized Expr and ExprContext related codes
hello-stephen commented on PR #16136: URL: https://github.com/apache/doris/pull/16136#issuecomment-1400534522 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.05 seconds load time: 491 seconds storage size: 17122202122 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230123152509_clickbench_pr_85049.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16133: [fix](icebergv2) fix bug that delete file reader is not opened
yiguolei merged PR #16133: URL: https://github.com/apache/doris/pull/16133 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16136: [refactor](remove expr) remove non vectorized Expr and ExprContext related codes
yiguolei merged PR #16136: URL: https://github.com/apache/doris/pull/16136 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei opened a new pull request, #16137: [refactor](remove unused code) remove storage buffer and orc reader
yiguolei opened a new pull request, #16137: URL: https://github.com/apache/doris/pull/16137 # Proposed changes 1. remove olap storage byte buffer 2. remove orc reader 3. remove time operator 4. remove read_write_util 5. remove aggregate funcs 6. remove compress.h and cpp 7. remove bhp_lib ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 5. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 8. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 9. Does it need to update dependencies: - [ ] Yes - [ ] No 10. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on a diff in pull request #15966: [Feature](map)support complex struct for doris
amorynan commented on code in PR #15966:
URL: https://github.com/apache/doris/pull/15966#discussion_r1084803369
##
be/src/olap/rowset/segment_v2/column_writer.cpp:
##
@@ -173,6 +173,119 @@ Status ColumnWriter::create(const ColumnWriterOptions&
opts, const TabletColumn*
*writer = std::move(writer_local);
return Status::OK();
}
+ case FieldType::OLAP_FIELD_TYPE_MAP: {
+DCHECK(column->get_subtype_count() == 2);
+// todo . here key and value is array only?
+const TabletColumn& key_column = column->get_sub_column(0); //
field_type is true key and value
+const TabletColumn& value_column = column->get_sub_column(1);
+
+// create null writer
+ScalarColumnWriter* null_writer = nullptr;
+if (opts.meta->is_nullable()) {
+FieldType null_type = FieldType::OLAP_FIELD_TYPE_TINYINT;
+ColumnWriterOptions null_options;
+null_options.meta = opts.meta->add_children_columns();
+null_options.meta->set_column_id(3);
+null_options.meta->set_unique_id(3);
+null_options.meta->set_type(null_type);
+null_options.meta->set_is_nullable(false);
+null_options.meta->set_length(
+
get_scalar_type_info()->size());
+null_options.meta->set_encoding(DEFAULT_ENCODING);
+null_options.meta->set_compression(opts.meta->compression());
+
+null_options.need_zone_map = false;
+null_options.need_bloom_filter = false;
+null_options.need_bitmap_index = false;
+
+TabletColumn null_column = TabletColumn(
+OLAP_FIELD_AGGREGATION_NONE, null_type, false,
+null_options.meta->unique_id(),
null_options.meta->length());
+null_column.set_name("nullable");
+null_column.set_index_length(-1); // no short key index
+std::unique_ptr
null_field(FieldFactory::create(null_column));
+null_writer =
+new ScalarColumnWriter(null_options,
std::move(null_field), file_writer);
+}
+
+// create key writer
+std::unique_ptr key_writer;
+ ColumnWriterOptions key_opts;
+TabletColumn key_list_column(OLAP_FIELD_AGGREGATION_NONE,
OLAP_FIELD_TYPE_ARRAY);
+{
+
key_list_column.add_sub_column(const_cast(key_column));
+//key_list_column.add_sub_column(key_column);
+key_list_column.set_name("map.key");
+key_list_column.set_index_length(-1);
+
+
+key_opts.meta = opts.meta->mutable_children_columns(0);
+key_opts.meta->set_column_id(4);
+key_opts.meta->set_unique_id(4);
+key_opts.meta->set_type(OLAP_FIELD_TYPE_ARRAY);
+key_opts.meta->set_length(0);
+ key_opts.meta->set_encoding(BIT_SHUFFLE);
+
key_opts.meta->set_compression(segment_v2::CompressionTypePB::LZ4F);
+key_opts.need_zone_map = false;
+ // no need key array's null map
+key_opts.meta->set_is_nullable(false);
+
+ColumnMetaPB* child_meta =
key_opts.meta->add_children_columns();
Review Comment:
I can use opts.meta directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on a diff in pull request #15966: [Feature](map)support complex struct for doris
amorynan commented on code in PR #15966:
URL: https://github.com/apache/doris/pull/15966#discussion_r1084803369
##
be/src/olap/rowset/segment_v2/column_writer.cpp:
##
@@ -173,6 +173,119 @@ Status ColumnWriter::create(const ColumnWriterOptions&
opts, const TabletColumn*
*writer = std::move(writer_local);
return Status::OK();
}
+ case FieldType::OLAP_FIELD_TYPE_MAP: {
+DCHECK(column->get_subtype_count() == 2);
+// todo . here key and value is array only?
+const TabletColumn& key_column = column->get_sub_column(0); //
field_type is true key and value
+const TabletColumn& value_column = column->get_sub_column(1);
+
+// create null writer
+ScalarColumnWriter* null_writer = nullptr;
+if (opts.meta->is_nullable()) {
+FieldType null_type = FieldType::OLAP_FIELD_TYPE_TINYINT;
+ColumnWriterOptions null_options;
+null_options.meta = opts.meta->add_children_columns();
+null_options.meta->set_column_id(3);
+null_options.meta->set_unique_id(3);
+null_options.meta->set_type(null_type);
+null_options.meta->set_is_nullable(false);
+null_options.meta->set_length(
+
get_scalar_type_info()->size());
+null_options.meta->set_encoding(DEFAULT_ENCODING);
+null_options.meta->set_compression(opts.meta->compression());
+
+null_options.need_zone_map = false;
+null_options.need_bloom_filter = false;
+null_options.need_bitmap_index = false;
+
+TabletColumn null_column = TabletColumn(
+OLAP_FIELD_AGGREGATION_NONE, null_type, false,
+null_options.meta->unique_id(),
null_options.meta->length());
+null_column.set_name("nullable");
+null_column.set_index_length(-1); // no short key index
+std::unique_ptr
null_field(FieldFactory::create(null_column));
+null_writer =
+new ScalarColumnWriter(null_options,
std::move(null_field), file_writer);
+}
+
+// create key writer
+std::unique_ptr key_writer;
+ ColumnWriterOptions key_opts;
+TabletColumn key_list_column(OLAP_FIELD_AGGREGATION_NONE,
OLAP_FIELD_TYPE_ARRAY);
+{
+
key_list_column.add_sub_column(const_cast(key_column));
+//key_list_column.add_sub_column(key_column);
+key_list_column.set_name("map.key");
+key_list_column.set_index_length(-1);
+
+
+key_opts.meta = opts.meta->mutable_children_columns(0);
+key_opts.meta->set_column_id(4);
+key_opts.meta->set_unique_id(4);
+key_opts.meta->set_type(OLAP_FIELD_TYPE_ARRAY);
+key_opts.meta->set_length(0);
+ key_opts.meta->set_encoding(BIT_SHUFFLE);
+
key_opts.meta->set_compression(segment_v2::CompressionTypePB::LZ4F);
+key_opts.need_zone_map = false;
+ // no need key array's null map
+key_opts.meta->set_is_nullable(false);
+
+ColumnMetaPB* child_meta =
key_opts.meta->add_children_columns();
Review Comment:
maybe I can use opts.meta directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #15839: [Feature] support segment builder tool
morningman commented on PR #15839: URL: https://github.com/apache/doris/pull/15839#issuecomment-1401381547 Some questions and suggestions: 1. `builder_scanner` doesn't seem to be used? Only `builder_scanner_memtable` is used? 2. Need to unify the inputs and outputs: 1. Inputs: * header file in json and data file in parquet(can be orc or other supported file format) * In the code, the reading methods of different file systems can be unified, and there is no need to use `isHDFS` to judge. 2. Outputs * new header file in json and Doris segment data file 3. The final upload logic can be encapsulated without being limited to HDFS 4. I think we can generate a manifest file to save all output file list. So that the downstream system can read it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16137: [refactor](remove unused code) remove storage buffer and orc reader
hello-stephen commented on PR #16137: URL: https://github.com/apache/doris/pull/16137#issuecomment-1401390539 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.45 seconds load time: 533 seconds storage size: 17122754338 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230124044525_clickbench_pr_85076.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on a diff in pull request #15966: [Feature](map)support complex struct for doris
amorynan commented on code in PR #15966:
URL: https://github.com/apache/doris/pull/15966#discussion_r1084840982
##
be/src/olap/types.cpp:
##
@@ -218,6 +233,11 @@ TypeInfoPtr get_type_info(const TabletColumn* col) {
TypeInfoPtr clone_type_info(const TypeInfo* type_info) {
if (is_scalar_type(type_info->type())) {
return create_static_type_info_ptr(type_info);
+} else if (type_info->type() == OLAP_FIELD_TYPE_MAP) {
+const auto map_type_info = dynamic_cast(type_info);
+return create_dynamic_type_info_ptr(
+new
MapTypeInfo(clone_type_info(map_type_info->get_key_type_info()),
+
clone_type_info(map_type_info->get_value_type_info(;
} else {
Review Comment:
so no default type or else ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on a diff in pull request #15966: [Feature](map)support complex struct for doris
amorynan commented on code in PR #15966:
URL: https://github.com/apache/doris/pull/15966#discussion_r1084955323
##
fe/fe-core/src/main/java/org/apache/doris/planner/SetOperationNode.java:
##
@@ -185,6 +185,9 @@ public void finalize(Analyzer analyzer) throws
UserException {
// corresponding output slot isn't being materialized)
materializedResultExprLists.clear();
Preconditions.checkState(resultExprLists.size() == children.size());
+if (analyzer.getDescTbl().getTupleDesc(tupleId) == null) {
Review Comment:
I met a npe when I make unitest in
`List slots =
analyzer.getDescTbl().getTupleDesc(tupleId).getSlots();`
because analyzer.getDescTbl().getTupleDesc(tupleId) return null...
so I add this, maybe it is not important , and it should not fix in this
pr. I will delete this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on a diff in pull request #15966: [Feature](map)support complex struct for doris
amorynan commented on code in PR #15966:
URL: https://github.com/apache/doris/pull/15966#discussion_r1084963919
##
fe/fe-core/src/main/java/org/apache/doris/catalog/MapType.java:
##
@@ -98,4 +134,16 @@ public void toThrift(TTypeDesc container) {
keyType.toThrift(container);
valueType.toThrift(container);
}
+
+@Override
+public TColumnType toColumnTypeThrift() {
+TColumnType thrift = new TColumnType();
+thrift.type = PrimitiveType.MAP.toThrift();
Review Comment:
Yes, I think this is only map column declare . And I refer to ArrayType
which not process its nested child column type too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on a diff in pull request #15966: [Feature](map)support complex struct for doris
amorynan commented on code in PR #15966:
URL: https://github.com/apache/doris/pull/15966#discussion_r1084974457
##
fe/fe-core/src/main/java/org/apache/doris/analysis/MapLiteral.java:
##
@@ -0,0 +1,179 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.MapType;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.thrift.TExprNode;
+import org.apache.doris.thrift.TExprNodeType;
+import org.apache.doris.thrift.TTypeDesc;
+import org.apache.doris.thrift.TTypeNode;
+
+import org.apache.commons.lang.StringUtils;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+
+// INSERT INTO table_map VALUES ({'key1':1, 'key2':10, 'k3':100}),
({'key1':2,'key2':20}), ({'key1':3,'key2':30});
+// MapLiteral is one row-based literal
+public class MapLiteral extends LiteralExpr {
+
+public MapLiteral() {
+type = new MapType(Type.NULL, Type.NULL);
+children = new ArrayList<>();
+}
+
+public MapLiteral(LiteralExpr... exprs) throws AnalysisException {
+Type keyType = Type.NULL;
+Type valueType = Type.NULL;
+children = new ArrayList<>();
+int idx = 0;
+for (LiteralExpr expr : exprs) {
+if (idx % 2 == 0) {
+if (keyType == Type.NULL) {
+keyType = expr.getType();
+} else {
+keyType = Type.getAssignmentCompatibleType(keyType,
expr.getType(), false);
+}
+if (keyType == Type.INVALID) {
+throw new AnalysisException("Invalid element type in Map");
+}
+} else {
+if (valueType == Type.NULL) {
+valueType = expr.getType();
+} else {
+valueType = Type.getAssignmentCompatibleType(valueType,
expr.getType(), false);
+}
+if (valueType == Type.INVALID) {
+throw new AnalysisException("Invalid element type in Map");
+}
+}
+children.add(expr);
+++ idx;
+}
+
+type = new MapType(keyType, valueType);
+}
+
+protected MapLiteral(MapLiteral other) {
+super(other);
+}
+
+@Override
+public Expr uncheckedCastTo(Type targetType) throws AnalysisException {
+if (!targetType.isMapType()) {
+return super.uncheckedCastTo(targetType);
+}
+MapLiteral literal = new MapLiteral(this);
+Type keyType = ((MapType) targetType).getKeyType();
+Type valueType = ((MapType) targetType).getValueType();
+
+for (int i = 0; i < children.size(); ++ i) {
+Expr child = children.get(i);
+if ((i & 1) == 0) {
+literal.children.set(i, child.uncheckedCastTo(keyType));
+} else {
+literal.children.set(i, child.uncheckedCastTo(valueType));
+}
+}
+literal.setType(targetType);
+return literal;
+}
+
+@Override
+public void checkValueValid() throws AnalysisException {
+for (Expr e : children) {
+e.checkValueValid();
+}
+}
+
+@Override
+protected String toSqlImpl() {
+List list = new ArrayList<>(children.size());
+for (int i = 0; i < children.size(); i += 2) {
+list.add(children.get(i).toSqlImpl() + ":" + children.get(i +
1).toSqlImpl());
+}
+return "MAP{" + StringUtils.join(list, ", ") + "}";
+}
+
+@Override
+protected void toThrift(TExprNode msg) {
+msg.node_type = TExprNodeType.MAP_LITERAL;
+TTypeDesc container = new TTypeDesc();
+container.setTypes(new ArrayList());
+type.toThrift(container);
+msg.setType(container);
+}
+
+@Override
+public Expr clone() {
+return new MapLiteral(this);
+}
+
+@Override
+public boolean isMinValue() {
+return false;
+}
+
+@Override
+public int compareLiteral(LiteralExpr
[GitHub] [doris] amorynan commented on a diff in pull request #15966: [Feature](map)support complex struct for doris
amorynan commented on code in PR #15966:
URL: https://github.com/apache/doris/pull/15966#discussion_r1085058385
##
be/src/vec/functions/array/function_array_element.h:
##
@@ -81,6 +94,79 @@ class FunctionArrayElement : public IFunction {
}
private:
+//=== map element===//
+ColumnPtr _get_mapped_idx(const ColumnArray& key_column,
+ const ColumnWithTypeAndName& argument) {
+if (key_column.get_data().is_column_string()) {
+return _mapped_key_string(key_column, argument);
+}
+return nullptr;
+}
+
+ColumnPtr _get_mapped_value(const ColumnArray& val_column,
Review Comment:
deleted !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on a diff in pull request #15966: [Feature](map)support complex struct for doris
amorynan commented on code in PR #15966:
URL: https://github.com/apache/doris/pull/15966#discussion_r1085068246
##
be/src/vec/functions/array/function_array_element.h:
##
@@ -81,6 +94,79 @@ class FunctionArrayElement : public IFunction {
}
private:
+//=== map element===//
+ColumnPtr _get_mapped_idx(const ColumnArray& key_column,
+ const ColumnWithTypeAndName& argument) {
+if (key_column.get_data().is_column_string()) {
+return _mapped_key_string(key_column, argument);
+}
+return nullptr;
+}
+
+ColumnPtr _get_mapped_value(const ColumnArray& val_column,
+const IColumn& matched_indices,
+const UInt8* src_null_map,
+UInt8* dst_null_map) {
+const UInt8* nested_null_map = nullptr;
+ColumnPtr nested_column = nullptr;
+if (is_column_nullable(val_column.get_data())) {
+const auto& nested_null_column =
+reinterpret_cast(val_column.get_data());
+nested_null_map =
nested_null_column.get_null_map_column().get_data().data();
+nested_column = nested_null_column.get_nested_column_ptr();
+} else {
+nested_column = val_column.get_data_ptr();
+}
+if (check_column(nested_column)) {
+return _execute_number(val_column.get_offsets(),
*nested_column,
+src_null_map, matched_indices,
+nested_null_map, dst_null_map);
+} else if (check_column(nested_column)) {
+_execute_number(val_column.get_offsets(),
*nested_column,
+src_null_map, matched_indices,
+nested_null_map, dst_null_map);
+}
+return nullptr;
+}
+
+ColumnPtr _mapped_key_string(const ColumnArray& column,
+const ColumnWithTypeAndName& argument) {
+auto right_column = argument.column->convert_to_full_column_if_const();
+const ColumnString& match_key = reinterpret_cast(*right_column);
+const ColumnArray::Offsets64& offsets = column.get_offsets();
+ColumnPtr nested_ptr = nullptr;
+if (is_column_nullable(column.get_data())) {
+nested_ptr = reinterpret_cast(column.get_data()).get_nested_column_ptr();
+} else {
+nested_ptr = column.get_data_ptr();
+}
+const ColumnString& nested_key = reinterpret_cast(*nested_ptr);
+size_t rows = offsets.size();
+// prepare return data
+auto matched_indices = ColumnVector::create();
+matched_indices->reserve(rows);
+
+for (size_t i = 0; i < rows; i++)
+{
+bool matched = false;
+size_t begin = offsets[i - 1];
+size_t end = offsets[i];
+for (size_t j = begin; j < end; j++) {
+if (nested_key.get_data_at(j) == match_key.get_data_at(i)) {
Review Comment:
done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #16135: [fix](icebergv2) fix icebergv2 delete file open bug and column case insensitive
morningman merged PR #16135: URL: https://github.com/apache/doris/pull/16135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16137: [refactor](remove unused code) remove storage buffer and orc reader
yiguolei merged PR #16137: URL: https://github.com/apache/doris/pull/16137 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #16111: [fix](multi catalog)Support parquet and orc upper case column name.
morningman commented on code in PR #16111:
URL: https://github.com/apache/doris/pull/16111#discussion_r1085449722
##
be/src/vec/exec/format/orc/vorc_reader.cpp:
##
@@ -780,7 +791,9 @@ Status OrcReader::get_next_block(Block* block, size_t*
read_rows, bool* eof) {
}
}
const auto& batch_vec =
down_cast(_batch.get())->fields;
-for (auto& col : _read_cols) {
+for (auto& origin_col : _read_cols) {
+std::string col = origin_col;
+transform(col.begin(), col.end(), col.begin(), ::tolower);
Review Comment:
save `lower case _read_cols` so that we don't need to transform it for each
block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei opened a new pull request, #16138: [refactor](remove non vec code) remove non vec functions and AggregateInfo
yiguolei opened a new pull request, #16138: URL: https://github.com/apache/doris/pull/16138 # Proposed changes remove these functions: CastFunctions EncryptionFunctions TimestampFunctions UtilityFunctions BitmapFunctions QuantileStateFunctions HashFunctions TopNFunctions ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #16123: [enhancement-wip](BE http)Support BE http service with brpc
morningman commented on PR #16123: URL: https://github.com/apache/doris/pull/16123#issuecomment-1402115367 Is there a way to keep http port unchanged but using brpc implemented http server? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener merged pull request #15811: [doc](cold hot separation)cold hot separation document adjustment
jackwener merged PR #15811: URL: https://github.com/apache/doris/pull/15811 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16138: [refactor](remove non vec code) remove non vec functions and AggregateInfo
hello-stephen commented on PR #16138: URL: https://github.com/apache/doris/pull/16138#issuecomment-1402184200 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.2 seconds load time: 491 seconds storage size: 17121772789 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230124155713_clickbench_pr_85098.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ByteYue opened a new issue, #16139: [Bug] vertical compaction segment writer's rows count is not consistent
ByteYue opened a new issue, #16139: URL: https://github.com/apache/doris/issues/16139 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version commit id: d062ca2944763d7f9629a5eb20822dcfbe3f43d0 compiled with asan using clang ### What's Wrong? BE would crash when trying to do vertical compaction. After searching the log i found out that it was all about the row_count and _num_rows_written check assert as follows.   Then i searched the tablet doing vertical compaction using the tablet id in BE.INFO log.  It is owned by table delete_sign_test.  And i see that this table is with one `__DORIS_DELETE_SIGN__ `.   Then i think this bug is due to the inconsistent rows nums with key and value column. ### What You Expected? Do the vertical compaction successfully. ### How to Reproduce? First, you need to type all the stmts inside `delete_p0/test_delete_sign_unique.sql`, then you have to wait for the compaction task to begin... I know it sounds stupid, i tried to trigger the compaction manually using compaction http action then i failed cause the version didn't meet the requirement. ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16138: [refactor](remove non vec code) remove non vec functions and AggregateInfo
yiguolei merged PR #16138: URL: https://github.com/apache/doris/pull/16138 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu commented on a diff in pull request #15128: [Feature](export) Support cancel export statement
dutyu commented on code in PR #15128:
URL: https://github.com/apache/doris/pull/15128#discussion_r1086250497
##
fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java:
##
@@ -294,12 +351,12 @@ public void replayCreateExportJob(ExportJob job) {
}
public void replayUpdateJobState(long jobId, ExportJob.JobState newState) {
-writeLock();
+readLock();
Review Comment:
This lock is designed to protect `idToJob` and `labelToJobId`,
`replayUpdateJobState` method just read from the `idToJob`, so use a read lock
may be more reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on a diff in pull request #15966: [Feature](map)support complex struct for doris
amorynan commented on code in PR #15966:
URL: https://github.com/apache/doris/pull/15966#discussion_r1086258307
##
be/src/vec/olap/olap_data_convertor.cpp:
##
@@ -712,4 +723,79 @@ Status
OlapBlockDataConvertor::OlapColumnDataConvertorArray::convert_to_olap(
return Status::OK();
}
+Status OlapBlockDataConvertor::OlapColumnDataConvertorMap::convert_to_olap() {
+const ColumnMap* column_map = nullptr;
+const DataTypeMap* data_type_map = nullptr;
+if (_nullmap) {
+const auto* nullable_column =
+assert_cast(_typed_column.column.get());
+column_map =
+assert_cast(nullable_column->get_nested_column_ptr().get());
+data_type_map = assert_cast(
+(assert_cast(_typed_column.type.get())->get_nested_type())
+.get());
+} else {
+column_map = assert_cast(_typed_column.column.get());
+data_type_map = assert_cast(_typed_column.type.get());
+}
+assert(column_map);
+assert(data_type_map);
+
+return convert_to_olap(column_map, data_type_map);
+}
+
+Status OlapBlockDataConvertor::OlapColumnDataConvertorMap::convert_to_olap(
+const ColumnMap* column_map,
+const DataTypeMap* data_type_map) {
+
+ColumnPtr key_data = column_map->get_keys_ptr();
+ColumnPtr value_data = column_map->get_values_ptr();
+if (column_map->get_keys().is_nullable()) {
+const auto& key_nullable_column =
+assert_cast(column_map->get_keys());
+key_data = key_nullable_column.get_nested_column_ptr();
+}
+
+if (column_map->get_values().is_nullable()) {
+const auto& val_nullable_column =
+assert_cast(column_map->get_values());
+value_data = val_nullable_column.get_nested_column_ptr();
+}
+
+const auto& offsets = column_map->get_offsets(); // use keys offsets
+
+ColumnWithTypeAndName key_typed_column = {
+key_data, remove_nullable(data_type_map->get_keys()),"map.key"};
+_key_convertor->set_source_column(key_typed_column, _row_pos, _num_rows);
+_key_convertor->convert_to_olap();
+
+ColumnWithTypeAndName value_typed_column = {
+value_data, remove_nullable(data_type_map->get_values()),
"map.value"};
+_value_convertor->set_source_column(value_typed_column, _row_pos,
_num_rows);
+_value_convertor->convert_to_olap();
+
+MapValue* map_value = _values.data();
+for (size_t i = 0; i < _num_rows; ++i, ++map_value) {
+int64_t cur_pos = _row_pos + i;
+int64_t prev_pos = cur_pos - 1;
+if (_nullmap && _nullmap[cur_pos]) {
+continue;
+}
+auto single_map_size = offsets[cur_pos] - offsets[prev_pos];
+new (map_value) MapValue(single_map_size);
Review Comment:
? You mean one MapValue contains more than one rows ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on a diff in pull request #15966: [Feature](map)support complex struct for doris
amorynan commented on code in PR #15966:
URL: https://github.com/apache/doris/pull/15966#discussion_r1086264303
##
be/src/vec/functions/array/function_array_element.h:
##
@@ -44,12 +46,18 @@ class FunctionArrayElement : public IFunction {
size_t get_number_of_arguments() const override { return 2; }
DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
-DCHECK(is_array(arguments[0]))
-<< "first argument for function: " << name << " should be
DataTypeArray";
-DCHECK(is_integer(arguments[1]))
-<< "second argument for function: " << name << " should be
Integer";
-return make_nullable(
-
check_and_get_data_type(arguments[0].get())->get_nested_type());
+DCHECK(is_array(arguments[0]) || is_map(arguments[0]))
+<< "first argument for function: " << name << " should be
DataTypeArray or DataTypeMap";
+if (is_array(arguments[0])) {
+DCHECK(is_integer(arguments[1])) << "second argument for function:
" << name << " should be Integer for array element";
+return make_nullable(
+
check_and_get_data_type(arguments[0].get())->get_nested_type());
+} else {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ByteYue commented on a diff in pull request #16107: [Chore](EditLog) refine error output on edit log
ByteYue commented on code in PR #16107:
URL: https://github.com/apache/doris/pull/16107#discussion_r1086313255
##
fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java:
##
@@ -994,9 +994,10 @@ public static void loadJournal(Env env, JournalEntity
journal) {
* log a warning here to debug when happens. This could happen to
other meta
* like DB.
*/
-LOG.warn("[INCONSISTENT META] replay failed {}: {}", journal,
e.getMessage(), e);
+LOG.warn("[INCONSISTENT META] replay failed {}: {}", journal,
e.getMessage());
} catch (Exception e) {
-LOG.error("Operation Type {}", opCode, e);
+LOG.error("Operation Type {}: {}", opCode, e.getMessage());
+e.printStackTrace();
Review Comment:
Using `printStackTrace` would print the error msg to the stdout, i think it
wouldn't help us debug. We can just log e so it will return the whole stack. If
you are worried about the `Fast Throw` optimization that jvm would do, you can
add `-XX:-OmitStackTraceInFastThrow` in start-fe.sh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #16123: [enhancement-wip](BE http)Support BE http service with brpc
TangSiyang2001 commented on PR #16123: URL: https://github.com/apache/doris/pull/16123#issuecomment-1403377434 > Is there a way to keep http port unchanged but using brpc implemented http server? Yes, doris has now got a brpc port for internal service and has established a brpc server. Becase brpc server supports several services ,we can add the new http service to this server to share the brpc port, and the original http port is completely unaffected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui opened a new pull request, #16140: [fix](metric) fix be down when enable_system_metrics is false
caiconghui opened a new pull request, #16140: URL: https://github.com/apache/doris/pull/16140 # Proposed changes Issue Number: close #xxx ## Problem summary if we set enable_system_metrics to false, we will see be down with following message "enable metric calculator failed, maybe you set enable_system_metrics to false ", so fix it ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16140: [fix](metric) fix be down when enable_system_metrics is false
hello-stephen commented on PR #16140: URL: https://github.com/apache/doris/pull/16140#issuecomment-1403639411 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.23 seconds load time: 531 seconds storage size: 17122302421 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230125134146_clickbench_pr_85160.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zddr closed pull request #16091: [wip](auth)Implementing privilege management with rbac model
zddr closed pull request #16091: [wip](auth)Implementing privilege management with rbac model URL: https://github.com/apache/doris/pull/16091 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei opened a new pull request, #16141: [refactor](remove non vec code) remove json functions string functions match functions and some code
yiguolei opened a new pull request, #16141: URL: https://github.com/apache/doris/pull/16141 # Proposed changes 1. remove json functions code 2. remove string functions code 3. remove math functions code 4. move MatchPredicate to olap since it is only used in storage predicate process 5. remove some code in tuple, Tuple structure should be removed in the future. 6. remove many code in collection value structure, they are useless ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 7. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 8. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 9. Does it need to update dependencies: - [ ] Yes - [ ] No 10. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16141: [refactor](remove non vec code) remove json functions string functions match functions and some code
hello-stephen commented on PR #16141: URL: https://github.com/apache/doris/pull/16141#issuecomment-1404607050 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.87 seconds load time: 499 seconds storage size: 17119089161 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230126062719_clickbench_pr_85212.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhengshubin opened a new issue, #16142: [Bug] be dead when execute hive catalog dml
zhengshubin opened a new issue, #16142: URL: https://github.com/apache/doris/issues/16142 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version hive 3.1.3 doris 1.2.1 ### What's Wrong? I had create hive catalog like this: CREATE RESOURCE hms_resource PROPERTIES ( 'type'='hms', 'hive.metastore.uris' = 'thrift://ip:9083', 'hadoop.username' = 'web', 'dfs.nameservices'='cluster01', 'dfs.ha.namenodes.cluster01'='nn1,nn2', 'dfs.namenode.rpc-address.cluster01.nn1'='ip:8020', 'dfs.namenode.rpc-address.cluster01.nn2'='ip:8020', 'dfs.client.failover.proxy.provider.cluster01'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider' ); CREATE CATALOG hive WITH RESOURCE hms_resource; use hive.`default`; When I use like show tables,it work. But when I use dml like : SELECT count(1) FROM ms_3; The one of the be would dead. FE.WARN.log : 2023-01-26 14:36:15,089 WARN (heartbeat-mgr-pool-0|102) [HeartbeatMgr$BackendHeartbeatHandler.call():268] backend heartbeat got exception org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[spark-dpp-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[spark-dpp-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[spark-dpp-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[spark-dpp-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[spark-dpp-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[spark-dpp-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at org.apache.doris.thrift.HeartbeatService$Client.recvHeartbeat(HeartbeatService.java:61) ~[spark-dpp-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at org.apache.doris.thrift.HeartbeatService$Client.heartbeat(HeartbeatService.java:48) ~[spark-dpp-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:226) ~[doris-fe.jar:1.0-SNAPSHOT] at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:203) ~[doris-fe.jar:1.0-SNAPSHOT] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_282] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_282] Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_282] at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_282] at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_282] at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_282] at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_282] at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_282] at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_282] at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[spark-dpp-1.0-SNAPSHOT.jar:1.0-SNAPSHOT] ... 13 more 2023-01-26 14:36:15,092 WARN (heartbeat mgr|31) [HeartbeatMgr.runAfterCatalogReady():139] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.SocketTimeoutException: Read timed out, beId: 16006, beHost: 1.2.3.4, bePort: 0, httpPort: 0, brpcPort: 0 ### What You Expected? BE don't dead whether hive catalog dml execute successfully or not ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --
[GitHub] [doris] yiguolei merged pull request #16141: [refactor](remove non vec code) remove json functions string functions match functions and some code
yiguolei merged PR #16141: URL: https://github.com/apache/doris/pull/16141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yukang-Lian closed pull request #15937: [fix](array) fix array(datetimev2) and array(decimalv3) returning wrong result
Yukang-Lian closed pull request #15937: [fix](array) fix array(datetimev2) and array(decimalv3) returning wrong result URL: https://github.com/apache/doris/pull/15937 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SaintBacchus commented on a diff in pull request #15511: [feature](Load)Suppot MySQL Load Data
SaintBacchus commented on code in PR #15511:
URL: https://github.com/apache/doris/pull/15511#discussion_r1087610231
##
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##
@@ -1836,6 +1841,41 @@ private void handleExplainStmt(String result) throws
IOException {
context.getState().setEof();
}
+private void handleLoadStmt() {
+try {
+LoadStmt loadStmt = (LoadStmt) parsedStmt;
+EtlJobType jobType = loadStmt.getEtlJobType();
+if (jobType == EtlJobType.UNKNOWN) {
+throw new DdlException("Unknown load job type");
+}
+if (jobType == EtlJobType.HADOOP) {
+throw new DdlException("Load job by hadoop cluster is
disabled."
++ " Try using broker load. See 'help broker load;'");
+}
+LoadManager loadManager = context.getEnv().getLoadManager();
+if (jobType == EtlJobType.LOCAL_FILE) {
+if (!context.getCapability().isClientLocalFile()) {
+throw new DdlException("Doris server does not support load
local file from mysql client.");
Review Comment:
use a `1148` error code of these case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei opened a new pull request, #16143: [refactor](remove unused code) remove many xxxVal structure
yiguolei opened a new pull request, #16143: URL: https://github.com/apache/doris/pull/16143 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SaintBacchus commented on a diff in pull request #15511: [feature](Load)Suppot MySQL Load Data
SaintBacchus commented on code in PR #15511:
URL: https://github.com/apache/doris/pull/15511#discussion_r1087614952
##
fe/fe-common/src/main/java/org/apache/doris/common/io/ByteBufferNetworkInputStream.java:
##
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.io;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+public class ByteBufferNetworkInputStream extends InputStream {
+private ArrayBlockingQueue queue;
+private ByteArrayInputStream currentInputStream;
+private volatile boolean finished = false;
+private volatile boolean closed = false;
+
+public ByteBufferNetworkInputStream() {
+this(32);
+}
+
+public ByteBufferNetworkInputStream(int capacity) {
+this.queue = new ArrayBlockingQueue<>(capacity);
+}
+
+public void fillByteBuffer(ByteBuffer buffer) throws IOException,
InterruptedException {
+if (closed) {
+throw new IOException("Stream is already closed.");
+}
+ByteArrayInputStream inputStream = new
ByteArrayInputStream(buffer.array(), buffer.position(), buffer.limit());
+queue.offer(inputStream, 300, TimeUnit.SECONDS);
Review Comment:
It is an big time but not too big time for timeout balancing the load
success rate and the cluster overload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SaintBacchus commented on a diff in pull request #15511: [feature](Load)Suppot MySQL Load Data
SaintBacchus commented on code in PR #15511:
URL: https://github.com/apache/doris/pull/15511#discussion_r1087615762
##
fe/fe-common/src/main/java/org/apache/doris/common/io/ByteBufferNetworkInputStream.java:
##
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.io;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+public class ByteBufferNetworkInputStream extends InputStream {
+private ArrayBlockingQueue queue;
+private ByteArrayInputStream currentInputStream;
+private volatile boolean finished = false;
+private volatile boolean closed = false;
+
+public ByteBufferNetworkInputStream() {
+this(32);
+}
+
+public ByteBufferNetworkInputStream(int capacity) {
+this.queue = new ArrayBlockingQueue<>(capacity);
+}
+
+public void fillByteBuffer(ByteBuffer buffer) throws IOException,
InterruptedException {
+if (closed) {
+throw new IOException("Stream is already closed.");
+}
+ByteArrayInputStream inputStream = new
ByteArrayInputStream(buffer.array(), buffer.position(), buffer.limit());
+queue.offer(inputStream, 300, TimeUnit.SECONDS);
Review Comment:
It's by experience
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yukang-Lian closed pull request #16014: [fix](agg-function) fix some agg-functions returning wrong results when the param type is array(datetimev2) and array(decimal)
Yukang-Lian closed pull request #16014: [fix](agg-function) fix some agg-functions returning wrong results when the param type is array(datetimev2) and array(decimal) URL: https://github.com/apache/doris/pull/16014 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chenlinzhong opened a new pull request, #16144: [fix](object) fix bitmap/hll data size
chenlinzhong opened a new pull request, #16144: URL: https://github.com/apache/doris/pull/16144 # Proposed changes Issue Number: close #xxx ## Problem summary bitmap/hll data size is wrong when length over 2G change type size_t -> int64_t Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chenlinzhong closed pull request #16144: [fix](object) fix bitmap/hll data size
chenlinzhong closed pull request #16144: [fix](object) fix bitmap/hll data size URL: https://github.com/apache/doris/pull/16144 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chenlinzhong closed pull request #16144: [fix](object) fix bitmap/hll data size
chenlinzhong closed pull request #16144: [fix](object) fix bitmap/hll data size URL: https://github.com/apache/doris/pull/16144 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16144: [fix](object) fix bitmap/hll data size
hello-stephen commented on PR #16144: URL: https://github.com/apache/doris/pull/16144#issuecomment-1406353623 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.68 seconds load time: 490 seconds storage size: 17123196490 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/2023012750_clickbench_pr_85275.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #16111: [fix](multi catalog)Support parquet and orc upper case column name.
morningman merged PR #16111: URL: https://github.com/apache/doris/pull/16111 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed issue #16023: [Enhancement] Optimize the position delete file filtering mechanism in iceberg v2.
morningman closed issue #16023: [Enhancement] Optimize the position delete file filtering mechanism in iceberg v2. URL: https://github.com/apache/doris/issues/16023 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #16024: [Enhancement](icebergv2) Optimize the position delete file filtering mechanism in iceberg v2 parquet reader.
morningman merged PR #16024: URL: https://github.com/apache/doris/pull/16024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #16107: [Chore](EditLog) refine error output on edit log
morningman commented on code in PR #16107:
URL: https://github.com/apache/doris/pull/16107#discussion_r1089138657
##
fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java:
##
@@ -994,9 +994,10 @@ public static void loadJournal(Env env, JournalEntity
journal) {
* log a warning here to debug when happens. This could happen to
other meta
* like DB.
*/
-LOG.warn("[INCONSISTENT META] replay failed {}: {}", journal,
e.getMessage(), e);
+LOG.warn("[INCONSISTENT META] replay failed {}: {}", journal,
e.getMessage());
} catch (Exception e) {
-LOG.error("Operation Type {}", opCode, e);
+LOG.error("Operation Type {}: {}", opCode, e.getMessage());
+e.printStackTrace();
Review Comment:
>
@ByteYue could you please submit a PR to add
`-XX:-OmitStackTraceInFastThrow`? I think this is necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #16140: [fix](metric) fix be down when set enable_system_metrics to false
morningman merged PR #16140: URL: https://github.com/apache/doris/pull/16140 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhengshengjun commented on pull request #16036: [feature] Support mutable property for partition
zhengshengjun commented on PR #16036: URL: https://github.com/apache/doris/pull/16036#issuecomment-1407246365 > Please add more details in your PR comment Hi, I have added details about this PR, could you please help me to review it ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16143: [refactor](remove unused code) remove many xxxVal structure
hello-stephen commented on PR #16143: URL: https://github.com/apache/doris/pull/16143#issuecomment-1407248353 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 39.59 seconds load time: 500 seconds storage size: 17122195373 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230128014357_clickbench_pr_85306.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] pengxiangyu closed pull request #14390: [Feature](remote)Remote Storage support single file when Cooldown.(FE)
pengxiangyu closed pull request #14390: [Feature](remote)Remote Storage support single file when Cooldown.(FE) URL: https://github.com/apache/doris/pull/14390 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #15972: [improvement](bitshuffle)Enable avx512 support in bitshuffle for performance boost
adonis0147 commented on PR #15972: URL: https://github.com/apache/doris/pull/15972#issuecomment-1407256849 Hi @HackToday , the Docker image has been updated. Please rebase the branch on master to re-trigger the checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
