Re: [PR] [enhance](io) Set segment file size to rowset meta [doris]
platoneko commented on PR #31853: URL: https://github.com/apache/doris/pull/31853#issuecomment-1980285293 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [nereids](topn-filter) support multi-topn filter (FE part) [doris]
englefly commented on PR #31485: URL: https://github.com/apache/doris/pull/31485#issuecomment-1980286483 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](nereids) support null safe eq runtime filter (FE part) [doris]
englefly commented on PR #31655: URL: https://github.com/apache/doris/pull/31655#issuecomment-1980289669 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](pipelineX) add mem control in local exchange sink [doris]
Mryange commented on PR #31840: URL: https://github.com/apache/doris/pull/31840#issuecomment-1980290044 run p0 10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](io) Set segment file size to rowset meta [doris]
platoneko commented on PR #31853: URL: https://github.com/apache/doris/pull/31853#issuecomment-1980290828 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature-WIP](partition) Suppor null range partition [doris]
zclllyybb commented on PR #31827: URL: https://github.com/apache/doris/pull/31827#issuecomment-1980294799 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) Fix wrong error msg when not in cloud mode [doris]
zclllyybb commented on PR #31814: URL: https://github.com/apache/doris/pull/31814#issuecomment-1980295162 > OlapScanNode.java 同样有一处漏了这个判断,方便的话可以帮一起修下 done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) Fix wrong error msg when not in cloud mode [doris]
zclllyybb commented on PR #31814: URL: https://github.com/apache/doris/pull/31814#issuecomment-1980295274 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[I] [Bug] Mysql bit 类型字段生成Doris schema 错误 [doris-flink-connector]
laizuan opened a new issue, #330: URL: https://github.com/apache/doris-flink-connector/issues/330 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version flink-doris-connector-1.18-1.5.2.jar flink: 1.18.1 mysql: 8.0 doris: 2.0.5 ### What's Wrong? 数据库字段`xxx`是bit类型,并且不为空。生成Doris预计如下: ```sql CREATE TABLE IF NOT EXISTS `xx`.`xx`( `id` BIGINT COMMENT '主键', `xxx` BOOLEAN DEFAULT 'b' 0 '' COMMENT 'xxx' ) UNIQUE KEY(`id`) DISTRIBUTED BY HASH(`id`) BUCKETS AUTO PROPERTIES ( 'replication_num' = '1', 'light_schema_change' = 'true' ) ``` 在控制执行SQL后的异常: ```txt Failed to execute sql: org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Syntax error in line 3: `xxx` BOOLEAN DEFAULT 'b' 0 '' COMMENT 'xxx' ^ Encountered: INTEGER LITERAL Expected: COMMA ``` ### What You Expected? ```sql CREATE TABLE IF NOT EXISTS `cm`.`xx`( `id` BIGINT COMMENT '主键', `xxx` BOOLEAN DEFAULT 'false' COMMENT 'xxx' ) UNIQUE KEY(`id`) DISTRIBUTED BY HASH(`id`) BUCKETS AUTO PROPERTIES ( 'replication_num' = '1', 'light_schema_change' = 'true' ) ``` ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [feature](Nereids): support make miss slot as null alias when converting anti join [doris]
XieJiann opened a new pull request, #31854: URL: https://github.com/apache/doris/pull/31854 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Nereids): support make miss slot as null alias when converting anti join [doris]
doris-robot commented on PR #31854: URL: https://github.com/apache/doris/pull/31854#issuecomment-1980306192 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature-WIP](partition) Suppor null range partition [doris]
HappenLee commented on code in PR #31827: URL: https://github.com/apache/doris/pull/31827#discussion_r1514005638 ## be/src/vec/sink/vrow_distribution.cpp: ## @@ -247,29 +256,34 @@ Status VRowDistribution::_generate_rows_distribution_for_auto_partition( if (!_missing_map.empty()) { // for missing partition keys, calc the missing partition and save in _partitions_need_create -auto [part_ctxs, part_funcs] = _get_partition_function(); -auto funcs_size = part_funcs.size(); +auto [part_ctxs, part_exprs] = _get_partition_function(); +auto part_col_num = part_exprs.size(); +// the two vectors are in column-first-order std::vector> col_strs; -col_strs.resize(funcs_size); - -for (int i = 0; i < funcs_size; ++i) { -auto return_type = part_funcs[i]->data_type(); -// expose the data column -vectorized::ColumnPtr range_left_col = -block->get_by_position(partition_cols_idx[i]).column; -if (const auto* nullable = - check_and_get_column(*range_left_col)) { -range_left_col = nullable->get_nested_column_ptr(); -return_type = assert_cast(return_type.get()) - ->get_nested_type(); +std::vector col_null_maps; +col_strs.resize(part_col_num); +col_null_maps.reserve(part_col_num); Review Comment: why not direct call reize ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Nereids): support make miss slot as null alias when converting anti join [doris]
XieJiann commented on PR #31854: URL: https://github.com/apache/doris/pull/31854#issuecomment-1980307569 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids): add struct info map in group [doris]
github-actions[bot] commented on PR #31800: URL: https://github.com/apache/doris/pull/31800#issuecomment-1980313529 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids): support null value in partition for updating [doris]
github-actions[bot] commented on PR #31843: URL: https://github.com/apache/doris/pull/31843#issuecomment-1980318184 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement][plsql] Support show procedure and show create procedure… [doris]
xinyiZzz commented on code in PR #31763: URL: https://github.com/apache/doris/pull/31763#discussion_r1513940706 ## fe/fe-core/src/main/java/org/apache/doris/plsql/functions/DorisFunctionRegistry.java: ## @@ -85,6 +89,37 @@ private String qualified(String name) { return (ConnectContext.get().getDatabase() + "." + name).toUpperCase(); } +@Override +public void showProcedure(List> columns) { +Map allProc = client.getAllPlsqlStoredProcedures(); +for (Map.Entry entry : allProc.entrySet()) { +List row = new ArrayList<>(); +PlsqlStoredProcedure proc = entry.getValue(); +row.add(proc.getName()); +row.add(Long.toString(proc.getCatalogId())); +row.add(Long.toString(proc.getDbId())); +row.add(proc.getPackageName()); +row.add(proc.getOwnerName()); +row.add(proc.getCreateTime()); +row.add(proc.getModifyTime()); +row.add(proc.getSource()); +columns.add(row); +} +} + +@Override +public void showCreateProcedure(FuncNameInfo procedureName, List> columns) { +Map allProc = client.getAllPlsqlStoredProcedures(); Review Comment: use `client.getPlsqlStoredProcedure(procedureName.getName(), procedureName.getCtlId(), procedureName.getDbId())` instead of `getAllPlsqlStoredProcedures` do not get all then check procedureName ## fe/fe-core/src/main/java/org/apache/doris/plsql/functions/DorisFunctionRegistry.java: ## @@ -196,10 +231,19 @@ public void addUserProcedure(Create_procedure_stmtContext ctx) { @Override public void save(FuncNameInfo procedureName, String source, boolean isForce) { try { +SimpleDateFormat f = new SimpleDateFormat("-MM-dd HH:mm:ss"); +String createTime = f.format(Calendar.getInstance().getTime()); +String modifyTime = f.format(Calendar.getInstance().getTime()); Review Comment: createTime and modifyTime may not be equal because get the current time separately `String modifyTime=createTime` seems better -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature] Support Array Type compare function [doris]
zclllyybb commented on code in PR #31701: URL: https://github.com/apache/doris/pull/31701#discussion_r1514019231 ## fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java: ## @@ -814,6 +814,9 @@ public TColumnType toColumnTypeThrift() { * here is to avoid special-casing logic in callers for concrete types. */ public boolean matchesType(Type t) { +if (this.isArrayType() && t.isArrayType()) { Review Comment: override in ArrayTypep ## fe/fe-common/src/main/java/org/apache/doris/catalog/Type.java: ## @@ -814,6 +814,9 @@ public TColumnType toColumnTypeThrift() { * here is to avoid special-casing logic in callers for concrete types. */ public boolean matchesType(Type t) { +if (this.isArrayType() && t.isArrayType()) { Review Comment: override in ArrayType -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](TrinoConnector-Catalog) Fe supports `trino-connector` catalog [doris]
BePPPower commented on code in PR #31768: URL: https://github.com/apache/doris/pull/31768#discussion_r1514019428 ## fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java: ## @@ -780,6 +795,31 @@ public static Env getCurrentEnv() { } } +private void initSpiEnvironment() { Review Comment: It will be modified in the next pr. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [chore](build) Using multithread to accelerate FE compilation [doris]
yagagagaga opened a new pull request, #31855: URL: https://github.com/apache/doris/pull/31855 ## Proposed changes Issue Number: close #xxx Maven 3.x has the capability to perform parallel builds. If enable parallel builds, 50% speed improvement is observed. before:  after:  ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](build) Using multithread to accelerate FE compilation [doris]
doris-robot commented on PR #31855: URL: https://github.com/apache/doris/pull/31855#issuecomment-1980323676 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](TrinoConnector-Catalog) Fe supports `trino-connector` catalog [doris]
github-actions[bot] commented on PR #31768: URL: https://github.com/apache/doris/pull/31768#issuecomment-1980324299 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](TrinoConnector-Catalog) Fe supports `trino-connector` catalog [doris]
github-actions[bot] commented on PR #31768: URL: https://github.com/apache/doris/pull/31768#issuecomment-1980324250 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](build) Using multithread to accelerate FE compilation [doris]
github-actions[bot] commented on PR #31855: URL: https://github.com/apache/doris/pull/31855#issuecomment-1980326610 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement][plsql] Support show procedure and show create procedure… [doris]
xinyiZzz commented on PR #31763: URL: https://github.com/apache/doris/pull/31763#issuecomment-1980326956 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](coordinator) should use fragment id not profile fragment id to cancel fragment [doris]
yiguolei commented on PR #31852: URL: https://github.com/apache/doris/pull/31852#issuecomment-1980334024 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [improvement](jdbc catalog) opt get db2 schema list & xml type mapping [doris]
zy-kkk opened a new pull request, #31856: URL: https://github.com/apache/doris/pull/31856 ## Proposed changes Issue Number: close #xxx 1. Trim Schema Names: Adapted the system to remove trailing spaces from DB2 schema names, ensuring compatibility without affecting query operations. 2. XML Mapping: Implemented a feature to directly map XML types to String. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](jdbc catalog) opt get db2 schema list & xml type mapping [doris]
doris-robot commented on PR #31856: URL: https://github.com/apache/doris/pull/31856#issuecomment-1980337227 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](jdbc catalog) opt get db2 schema list & xml type mapping [doris]
zy-kkk commented on PR #31856: URL: https://github.com/apache/doris/pull/31856#issuecomment-1980339333 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids): add struct info map in group [doris]
github-actions[bot] commented on PR #31800: URL: https://github.com/apache/doris/pull/31800#issuecomment-1980357263 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[I] [Bug] ky10 aarch64 mysql-connector-python cannot connect to FE [doris]
Bruceve1n opened a new issue, #31857: URL: https://github.com/apache/doris/issues/31857 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.2.4.1 $ md5sum apache-doris-1.2.4.1-bin-aarch64.tar.xz 888b211ad73a33c6a103e930c48f9135 *apache-doris-1.2.4.1-bin-aarch64.tar.xz ### What's Wrong? when I use mysql-connector-python 8.0.23 to 8.0.28 version, like py2.py3-none-any.whl package connect doris return Traceback (most recent call last): File "test.py", line 9, in cnx = mysql.connector.connect(**config) File "/usr/local/lib/python3.7/site-packages/mysql/connector/__init__.py", line 278, in connect return MySQLConnection(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/mysql/connector/connection.py", line 108, in __init__ self.connect(**kwargs) File "/usr/local/lib/python3.7/site-packages/mysql/connector/abstracts.py", line 1003, in connect self._open_connection() File "/usr/local/lib/python3.7/site-packages/mysql/connector/connection.py", line 355, in _open_connection self._ssl, self._conn_attrs) File "/usr/local/lib/python3.7/site-packages/mysql/connector/connection.py", line 220, in _do_auth self._auth_switch_request(username, password) File "/usr/local/lib/python3.7/site-packages/mysql/connector/connection.py", line 236, in _auth_switch_request packet = self._socket.recv() File "/usr/local/lib/python3.7/site-packages/mysql/connector/network.py", line 267, in recv_plain raise errors.InterfaceError(errno=2013) mysql.connector.errors.InterfaceError: 2013: Lost connection to MySQL server during query **but this script work fine on mysql 5.7** ### What You Expected? return 1 ### How to Reproduce? ### this is my linux Linux version 4.19.90-24.4.v2101.ky10.aarch64 ([KYLINSOFT@localhost.localdomain](mailto:%20KYLINSOFT@localhost.localdomain)) (gcc version 7.3.0 (GCC)) #1 SMP Mon May 24 14:45:37 CST 2021 ### this is my script import mysql.connector config = dict() config['user'] = 'root' config['password'] = 'password' config['host'] = 'ip' config['port'] = '9030' config['autocommit'] = True cnx = mysql.connector.connect(**config) cursor = cnx.cursor(dictionary=False, buffered=True) cursor.execute('select 1') result = cursor.fetchall() print(result) ### while doris work fine when I use mysql client login FE, show backends; all BE are alive ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](io) Set segment file size to rowset meta [doris]
gavinchou commented on code in PR #31853: URL: https://github.com/apache/doris/pull/31853#discussion_r1514051381 ## be/src/olap/rowset/beta_rowset_writer.cpp: ## @@ -664,6 +739,13 @@ Status BaseBetaRowsetWriter::_build_rowset_meta(RowsetMeta* rowset_meta, bool ch // TODO write zonemap to meta rowset_meta->set_empty((num_rows_written + _num_rows_written) == 0); rowset_meta->set_creation_time(time(nullptr)); + +if (auto seg_file_size = _seg_files.segments_file_size(); !seg_file_size.has_value()) { +LOG(ERROR) << seg_file_size.error(); Review Comment: Should we return an error status? ## be/src/olap/rowset/rowset_meta.h: ## @@ -314,6 +314,12 @@ class RowsetMeta { void set_txn_expiration(int64_t expiration) { _rowset_meta_pb.set_txn_expiration(expiration); } +// `seg_file_size` MUST ordered by segment id +void set_segments_file_size(const std::vector& seg_file_size); Review Comment: how to ensure > `seg_file_size` MUST ordered by segment id should we check the order or sort it in `set_segments_file_size` ## be/src/olap/rowset/beta_rowset_writer.cpp: ## @@ -96,12 +97,92 @@ void build_rowset_meta_with_spec_field(RowsetMeta& rowset_meta, } // namespace +SegmentFileCollection::~SegmentFileCollection() = default; + +Status SegmentFileCollection::add(int seg_id, io::FileWriterPtr&& writer) { +std::lock_guard lock(_lock); +if (_closed) [[unlikely]] { +DCHECK(false) << writer->path(); +return Status::InternalError("add to closed SegmentFileCollection"); +} + +_file_writers.emplace_back(seg_id, std::move(writer)); +return Status::OK(); +} + +Status SegmentFileCollection::close() { +{ +std::lock_guard lock(_lock); +if (_closed) [[unlikely]] { +DCHECK(false); +return Status::InternalError("double close SegmentFileCollection"); +} +_closed = true; +} + +for (auto&& [_, writer] : _file_writers) { +RETURN_IF_ERROR(writer->close()); +} + +return Status::OK(); +} + +Result> SegmentFileCollection::segments_file_size() { +std::lock_guard lock(_lock); +if (!_closed) [[unlikely]] { +DCHECK(false); +return ResultError(Status::InternalError("get segments file size without closed")); +} + +Status st; +std::vector seg_file_size(_file_writers.size(), 0); +bool succ = std::all_of(_file_writers.begin(), _file_writers.end(), [&](auto&& it) { +auto&& [seg_id, writer] = it; + +if (seg_id > seg_file_size.size()) [[unlikely]] { Review Comment: it seems to be `seg_id >= seg_file_size.size()` ? ## be/src/olap/rowset/vertical_beta_rowset_writer.h: ## @@ -17,28 +17,28 @@ #pragma once -#include -#include - #include +#include #include #include "common/status.h" #include "olap/rowset/beta_rowset_writer.h" #include "olap/rowset/segment_v2/segment_writer.h" -#include "olap/rowset/vertical_beta_rowset_writer_helper.h" namespace doris { namespace vectorized { class Block; } // namespace vectorized // for vertical compaction -// TODO(plat1ko): Inherited from template type `T`, `T` is `BetaRowsetWriter` or `CloudBetaRowsetWriter` -class VerticalBetaRowsetWriter final : public BetaRowsetWriter { +template +requires std::is_base_of_v +class VerticalBetaRowsetWriter final : public T { Review Comment: does gcc compile? ## be/src/olap/rowset/beta_rowset_writer.cpp: ## @@ -96,12 +97,92 @@ void build_rowset_meta_with_spec_field(RowsetMeta& rowset_meta, } // namespace +SegmentFileCollection::~SegmentFileCollection() = default; + +Status SegmentFileCollection::add(int seg_id, io::FileWriterPtr&& writer) { +std::lock_guard lock(_lock); +if (_closed) [[unlikely]] { +DCHECK(false) << writer->path(); +return Status::InternalError("add to closed SegmentFileCollection"); +} + +_file_writers.emplace_back(seg_id, std::move(writer)); +return Status::OK(); +} + +Status SegmentFileCollection::close() { +{ +std::lock_guard lock(_lock); +if (_closed) [[unlikely]] { +DCHECK(false); +return Status::InternalError("double close SegmentFileCollection"); +} +_closed = true; +} + +for (auto&& [_, writer] : _file_writers) { +RETURN_IF_ERROR(writer->close()); +} + +return Status::OK(); +} + +Result> SegmentFileCollection::segments_file_size() { +std::lock_guard lock(_lock); +if (!_closed) [[unlikely]] { +DCHECK(false); +return ResultError(Status::InternalError("get segments file size without closed")); +} + +Status st; +std::vector seg_file_size(_file_writers.size(), 0); +bool succ = std::all_of(_file_writers.begin(), _file_writers.end(), [&](auto&& it) { +auto&& [seg_id, writer] = it; + +if (seg_id > seg_file_size.size()) [[
Re: [PR] [enhancement](Nereids) refine and speedup analyzer [doris]
924060929 commented on PR #31792: URL: https://github.com/apache/doris/pull/31792#issuecomment-1980369001 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [feat](Nereids): add struct info map in group (#31800)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new dbccbe0fbba [feat](Nereids): add struct info map in group (#31800) dbccbe0fbba is described below commit dbccbe0fbbaaa6550eee7b3234c15910615c63a5 Author: 谢健 AuthorDate: Wed Mar 6 16:49:04 2024 +0800 [feat](Nereids): add struct info map in group (#31800) --- .../org/apache/doris/mtmv/MTMVRelationManager.java | 10 +- .../java/org/apache/doris/nereids/memo/Group.java | 7 +- .../apache/doris/nereids/memo/StructInfoMap.java | 123 + .../mv/InitMaterializationContextHook.java | 7 +- .../doris/nereids/memo/StructInfoMapTest.java | 81 ++ .../org/apache/doris/nereids/util/PlanChecker.java | 2 + .../apache/doris/utframe/TestWithFeService.java| 28 + 7 files changed, 252 insertions(+), 6 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java index aa7ffd2426d..723deaff740 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java @@ -35,6 +35,7 @@ import org.apache.doris.nereids.trees.plans.commands.info.TableNameInfo; import org.apache.doris.persist.AlterMTMV; import org.apache.doris.qe.ConnectContext; +import com.google.common.annotations.VisibleForTesting; import com.google.common.collect.ImmutableSet; import com.google.common.collect.Maps; import com.google.common.collect.Sets; @@ -71,8 +72,7 @@ public class MTMVRelationManager implements MTMVHookService { for (BaseTableInfo tableInfo : mvInfos) { try { MTMV mtmv = (MTMV) MTMVUtil.getTable(tableInfo); -if (!CollectionUtils - .isEmpty(MTMVRewriteUtil.getMTMVCanRewritePartitions(mtmv, ctx, System.currentTimeMillis( { +if (isMVPartitionValid(mtmv, ctx)) { res.add(mtmv); } } catch (AnalysisException e) { @@ -83,6 +83,12 @@ public class MTMVRelationManager implements MTMVHookService { return res; } +@VisibleForTesting +public boolean isMVPartitionValid(MTMV mtmv, ConnectContext ctx) { +return !CollectionUtils +.isEmpty(MTMVRewriteUtil.getMTMVCanRewritePartitions(mtmv, ctx, System.currentTimeMillis())); +} + private Set getMTMVInfos(List tableInfos) { Set mvInfos = Sets.newHashSet(); for (BaseTableInfo tableInfo : tableInfos) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java index 2a0e83c72a4..5a5abd56f95 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java @@ -19,7 +19,6 @@ package org.apache.doris.nereids.memo; import org.apache.doris.common.Pair; import org.apache.doris.nereids.cost.Cost; -import org.apache.doris.nereids.jobs.joinorder.hypergraph.HyperGraph; import org.apache.doris.nereids.properties.LogicalProperties; import org.apache.doris.nereids.properties.PhysicalProperties; import org.apache.doris.nereids.rules.exploration.mv.StructInfo; @@ -78,6 +77,8 @@ public class Group { private List structInfos = new ArrayList<>(); +private StructInfoMap structInfoMap = new StructInfoMap(); + /** * Constructor for Group. * @@ -418,8 +419,8 @@ public class Group { return false; } -public List getHyperGraphs() { -return new ArrayList<>(); +public StructInfoMap getstructInfoMap() { +return structInfoMap; } public boolean isProjectGroup() { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/StructInfoMap.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/StructInfoMap.java new file mode 100644 index 000..f7e09fd44d9 --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/StructInfoMap.java @@ -0,0 +1,123 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF A
Re: [PR] [feat](Nereids): add struct info map in group [doris]
jackwener merged PR #31800: URL: https://github.com/apache/doris/pull/31800 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [enhancement](Nereids): support more condition Date/DateTime Literal [doris]
jackwener opened a new pull request, #31858: URL: https://github.com/apache/doris/pull/31858 ## Proposed changes support ``` new DateTimeV2Literal("0-08-01 13:21:03"); new DateTimeV2Literal("0001-01-01: 00:01:01.001"); new DateTimeV2Literal("2021?01?01 00.00.00"); ``` ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](Nereids): support more condition Date/DateTime Literal [doris]
doris-robot commented on PR #31858: URL: https://github.com/apache/doris/pull/31858#issuecomment-1980382076 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] disable pipeline in cloud_p0 [doris]
dataroaring commented on PR #31825: URL: https://github.com/apache/doris/pull/31825#issuecomment-1980391035 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](pipelineX) support paritition tablet sink shuffle [doris]
zhangstar333 commented on PR #31689: URL: https://github.com/apache/doris/pull/31689#issuecomment-1980390311 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] Support follower sync query columns to master. [doris]
Jibing-Li opened a new pull request, #31859: URL: https://github.com/apache/doris/pull/31859 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Support follower sync query columns to master. [doris]
doris-robot commented on PR #31859: URL: https://github.com/apache/doris/pull/31859#issuecomment-1980393986 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Support follower sync query columns to master. [doris]
Jibing-Li merged PR #31859: URL: https://github.com/apache/doris/pull/31859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch high-priority-column updated: Support follower sync query columns to master. (#31859)
This is an automated email from the ASF dual-hosted git repository. lijibing pushed a commit to branch high-priority-column in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/high-priority-column by this push: new 14f606fbc97 Support follower sync query columns to master. (#31859) 14f606fbc97 is described below commit 14f606fbc977009123d30d9ff244fcea2a75e993 Author: Jibing-Li <64681310+jibing...@users.noreply.github.com> AuthorDate: Wed Mar 6 17:05:21 2024 +0800 Support follower sync query columns to master. (#31859) --- .../main/java/org/apache/doris/catalog/Env.java| 8 ++ .../apache/doris/service/FrontendServiceImpl.java | 8 ++ .../apache/doris/statistics/AnalysisManager.java | 15 +++ .../doris/statistics/FollowerColumnSender.java | 120 + .../doris/statistics/HighPriorityColumn.java | 11 ++ .../doris/statistics/StatisticsAutoCollector.java | 21 +++- gensrc/thrift/FrontendService.thrift | 13 +++ 7 files changed, 194 insertions(+), 2 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java index 823b7f18fda..3f34a6522dc 100755 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java @@ -240,6 +240,7 @@ import org.apache.doris.scheduler.registry.ExportTaskRegister; import org.apache.doris.service.ExecuteEnv; import org.apache.doris.service.FrontendOptions; import org.apache.doris.statistics.AnalysisManager; +import org.apache.doris.statistics.FollowerColumnSender; import org.apache.doris.statistics.StatisticsAutoCollector; import org.apache.doris.statistics.StatisticsCache; import org.apache.doris.statistics.StatisticsCleaner; @@ -519,6 +520,8 @@ public class Env { private StatisticsJobAppender statisticsJobAppender; +private FollowerColumnSender followerColumnSender; + private HiveTransactionMgr hiveTransactionMgr; private TopicPublisherThread topicPublisherThread; @@ -1719,6 +1722,11 @@ public class Env { if (analysisManager != null) { analysisManager.getStatisticsCache().preHeat(); } + +if (followerColumnSender == null) { +followerColumnSender = new FollowerColumnSender(); +followerColumnSender.start(); +} } // Set global variable 'lower_case_table_names' only when the cluster is initialized. diff --git a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java index f450af207bd..b9d74762d24 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java +++ b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java @@ -224,6 +224,7 @@ import org.apache.doris.thrift.TStreamLoadMultiTablePutResult; import org.apache.doris.thrift.TStreamLoadPutRequest; import org.apache.doris.thrift.TStreamLoadPutResult; import org.apache.doris.thrift.TStringLiteral; +import org.apache.doris.thrift.TSyncQueryColumns; import org.apache.doris.thrift.TTableIndexQueryStats; import org.apache.doris.thrift.TTableMetadataNameIds; import org.apache.doris.thrift.TTableQueryStats; @@ -3678,4 +3679,11 @@ public class FrontendServiceImpl implements FrontendService.Iface { return result; } +@Override +public TStatus syncQueryColumns(TSyncQueryColumns request) throws TException { + Env.getCurrentEnv().getAnalysisManager().mergeFollowerQueryColumns(request.highPriorityColumns, +request.midPriorityColumns); +return new TStatus(TStatusCode.OK); +} + } diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java index 0ff9e3a9e00..ed66eae0455 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java @@ -64,6 +64,7 @@ import org.apache.doris.statistics.util.StatisticsUtil; import org.apache.doris.system.Frontend; import org.apache.doris.system.SystemInfoService; import org.apache.doris.thrift.TInvalidateFollowerStatsCacheRequest; +import org.apache.doris.thrift.TQueryColumn; import com.google.common.annotations.VisibleForTesting; import com.google.common.collect.ImmutableList; @@ -1176,4 +1177,18 @@ public class AnalysisManager implements Writable { } } } + +public void mergeFollowerQueryColumns(Collection highColumns, +Collection midColumns) { +for (TQueryColumn c : highColumns) { +if (!highPriorityColumns.offer(new HighPriorityColumn(c.catalogId, c.dbId, c.tblId, c.colName))) { +break; +} +} +for (TQ
Re: [PR] [enhancement](Nereids): support more condition Date/DateTime Literal [doris]
jackwener commented on PR #31858: URL: https://github.com/apache/doris/pull/31858#issuecomment-1980395270 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](cloud) add CloudTabletStatMgr to capture stats in cloud mode [doris]
SWJTU-ZhangLei commented on code in PR #31818: URL: https://github.com/apache/doris/pull/31818#discussion_r1514088920 ## fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java: ## @@ -0,0 +1,273 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.catalog; + +import org.apache.doris.catalog.MaterializedIndex.IndexExtState; +import org.apache.doris.catalog.TableIf.TableType; +import org.apache.doris.cloud.proto.Cloud.GetTabletStatsRequest; +import org.apache.doris.cloud.proto.Cloud.GetTabletStatsResponse; +import org.apache.doris.cloud.proto.Cloud.MetaServiceCode; +import org.apache.doris.cloud.proto.Cloud.TabletIndexPB; +import org.apache.doris.cloud.proto.Cloud.TabletStatsPB; +import org.apache.doris.cloud.rpc.MetaServiceProxy; +import org.apache.doris.common.Config; +import org.apache.doris.common.Pair; +import org.apache.doris.common.util.MasterDaemon; +import org.apache.doris.rpc.RpcException; + +import lombok.Getter; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ForkJoinPool; + +/* + * CloudTabletStatMgr is for collecting tablet(replica) statistics from backends. + * Each FE will collect by itself. + */ +public class CloudTabletStatMgr extends MasterDaemon { +private static final Logger LOG = LogManager.getLogger(CloudTabletStatMgr.class); + +private ForkJoinPool taskPool = new ForkJoinPool(Runtime.getRuntime().availableProcessors()); Review Comment: The `taskPool` seems redundant -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [RuntimeFilter] (JDBC) jdbc scan node support IN/MIN_MAX runtime filter [doris]
DarvenDuan commented on PR #31741: URL: https://github.com/apache/doris/pull/31741#issuecomment-1980421316 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](partition) fix npe when prune partition with not exist partition column in mv [doris]
doris-robot commented on PR #31860: URL: https://github.com/apache/doris/pull/31860#issuecomment-1980432830 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Bug](partition) fix npe when prune partition with not exist partition column in mv [doris]
BiteThet opened a new pull request, #31860: URL: https://github.com/apache/doris/pull/31860 ## Proposed changes ```java 2024-03-06 16:13:46,503 WARN (mysql-nio-pool-0|440) [StmtExecutor.execute():515] Analyze failed. stmt[4, 50e64f616a9e4d42-b11d236f84051082] org.apache.doris.common.NereidsException: errCode = 2, detailMessage = at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:661) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:497) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:476) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:279) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:197) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:176) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:205) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:258) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:842) ~[?:?] 2024-03-06 16:13:46,503 WARN (mysql-nio-pool-0|440) [ConnectProcessor.handleQueryException():349] Process one query failed because. org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:661) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:497) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:476) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:279) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:197) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:176) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:205) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:258) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:842) ~[?:?] Caused by: java.lang.NullPointerException at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:903) ~[guava-32.1.2-jre.jar:?] at com.google.common.collect.SingletonImmutableSet.(SingletonImmutableSet.java:40) ~[guava-32.1.2-jre.jar:?] at com.google.common.collect.ImmutableSet.of(ImmutableSet.java:91) ~[guava-32.1.2-jre.jar:?] at com.google.common.collect.ImmutableSet.construct(ImmutableSet.java:201) ~[guava-32.1.2-jre.jar:?] at com.google.common.collect.ImmutableSet.constructUnknownDuplication(ImmutableSet.java:172) ~[guava-32.1.2-jre.jar:?] at com.google.common.collect.ImmutableSet.copyOf(ImmutableSet.java:245) ~[guava-32.1.2-jre.jar:?] at org.apache.doris.nereids.rules.expression.rules.PartitionPruner.prune(PartitionPruner.java:108) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.rules.rewrite.PruneOlapScanPartition.lambda$build$5(PruneOlapScanPartition.java:80) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.pattern.PatternMatcher$1.transform(PatternMatcher.java:92) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.jobs.rewrite.PlanTreeRewriteJob.rewrite(PlanTreeRewriteJob.java:54) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.jobs.rewrite.PlanTreeRewriteTopDownJob.execute(PlanTreeRewriteTopDownJob.java:48) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.jobs.scheduler.SimpleJobScheduler.executeJobPool(SimpleJobScheduler.java:43) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.jobs.rewrite.RootPlanTreeRewriteJob.execute(RootPlanTreeRewriteJob.java:54) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.jobs.executor.AbstractBatchJobExecutor.execut
[PR] [regress](spark)Add spark to read doris multiple data types cases [doris]
caoliang-web opened a new pull request, #31861: URL: https://github.com/apache/doris/pull/31861 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regress](spark)Add spark to read doris multiple data types cases [doris]
doris-robot commented on PR #31861: URL: https://github.com/apache/doris/pull/31861#issuecomment-1980438022 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](cloud) fix create tablet miss cluster key index [doris]
yujun777 opened a new pull request, #31862: URL: https://github.com/apache/doris/pull/31862 only base tablet + schema change tablet use cluster key index, other tablet don't. ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) fix create tablet miss cluster key index [doris]
doris-robot commented on PR #31862: URL: https://github.com/apache/doris/pull/31862#issuecomment-1980440376 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) fix create tablet miss cluster key index [doris]
yujun777 commented on PR #31862: URL: https://github.com/apache/doris/pull/31862#issuecomment-1980440555 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](schema change) Fix schema change fault when add complex type column [doris]
Yukang-Lian commented on PR #31824: URL: https://github.com/apache/doris/pull/31824#issuecomment-1980441544 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regression](subquery)Subquery test case not supported in add select list [doris]
caoliang-web commented on PR #31103: URL: https://github.com/apache/doris/pull/31103#issuecomment-1980441695 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Fix](Nereids) fix test distribute case return null pointer error [doris]
LiBinfeng-01 opened a new pull request, #31863: URL: https://github.com/apache/doris/pull/31863 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Nereids) fix test distribute case return null pointer error [doris]
doris-robot commented on PR #31863: URL: https://github.com/apache/doris/pull/31863#issuecomment-1980446015 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Nereids) fix test distribute case return null pointer error [doris]
LiBinfeng-01 commented on PR #31863: URL: https://github.com/apache/doris/pull/31863#issuecomment-1980446098 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) Fix FE cannot work when restarting after schema change run in cloud mode [doris]
Lchangliang commented on PR #31842: URL: https://github.com/apache/doris/pull/31842#issuecomment-1980450726 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] disable pipeline in cloud_p0 [doris]
doris-robot commented on PR #31825: URL: https://github.com/apache/doris/pull/31825#issuecomment-1980451659 TeamCity be ut coverage result: Function Coverage: 35.41% (8559/24174) Line Coverage: 27.16% (69471/255817) Region Coverage: 26.37% (36029/136641) Branch Coverage: 23.20% (18410/79342) Coverage Report: http://coverage.selectdb-in.cc/coverage/c891bb1668416ed3f8492901f182fa0438e25e73_c891bb1668416ed3f8492901f182fa0438e25e73/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [docs](docs)Add doris-kafka-connector usage documentation [doris]
zy-kkk merged PR #31848: URL: https://github.com/apache/doris/pull/31848 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [docs](kafka-connector) Add doris-kafka-connector usage documentation (#31729) (#31848)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new 4aa470c6682 [docs](kafka-connector) Add doris-kafka-connector usage documentation (#31729) (#31848) 4aa470c6682 is described below commit 4aa470c66822aeed79bb9e9cc63d2072f28784dd Author: wudongliang <46414265+donglian...@users.noreply.github.com> AuthorDate: Wed Mar 6 17:35:53 2024 +0800 [docs](kafka-connector) Add doris-kafka-connector usage documentation (#31729) (#31848) --- docs/en/docs/ecosystem/doris-kafka-connector.md| 192 docs/sidebars.json | 1 + docs/zh-CN/docs/ecosystem/doris-kafka-connector.md | 193 + 3 files changed, 386 insertions(+) diff --git a/docs/en/docs/ecosystem/doris-kafka-connector.md b/docs/en/docs/ecosystem/doris-kafka-connector.md new file mode 100644 index 000..bb0aa89e639 --- /dev/null +++ b/docs/en/docs/ecosystem/doris-kafka-connector.md @@ -0,0 +1,192 @@ +--- +{ +"title": "Doris Kafka Connector", +"language": "en" +} +--- + + + +[Kafka Connect](https://docs.confluent.io/platform/current/connect/index.html) is a scalable and reliable tool for data transmission between Apache Kafka and other systems. Connectors can be defined Move large amounts of data in and out of Kafka. + +Doris provides the Sink Connector plug-in, which can write data from Kafka topics to Doris. + +## Usage Doris Kafka Connector + +### Standalone mode startup + +Configure connect-standalone.properties + +```properties +# Modify broker address +bootstrap.servers=127.0.0.1:9092 +``` + +Configure doris-connector-sink.properties +Create doris-connector-sink.properties in the config directory and configure the following content: + +```properties +name=test-doris-sink +connector.class=org.apache.doris.kafka.connector.DorisSinkConnector +topics=topic_test +doris.topic2table.map=topic_test:test_kafka_tbl +buffer.count.records=1 +buffer.flush.time=120 +buffer.size.bytes=500 +doris.urls=10.10.10.1 +doris.http.port=8030 +doris.query.port=9030 +doris.user=root +doris.password= +doris.database=test_db +key.converter=org.apache.kafka.connect.storage.StringConverter +value.converter=org.apache.kafka.connect.json.JsonConverter +key.converter.schemas.enable=false +value.converter.schemas.enable=false +``` + +Start Standalone + +```shell +$KAFKA_HOME/bin/connect-standalone.sh -daemon $KAFKA_HOME/config/connect-standalone.properties $KAFKA_HOME/config/doris-connector-sink.properties +``` +:::warning +Note: It is generally not recommended to use standalone mode in a production environment. +::: + +### Distributed mode startup + +Configure connect-distributed.properties + +```properties +# Modify broker address +bootstrap.servers=127.0.0.1:9092 + +# Modify group.id, the same cluster needs to be consistent +group.id=connect-cluster +``` + + + +Start Distributed + +```shell +$KAFKA_HOME/bin/connect-distributed.sh -daemon $KAFKA_HOME/config/connect-distributed.properties +``` + + +Add Connector + +```shell +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"test-doris-sink-cluster", + "config":{ +"connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", +"topics":"topic_test", +"doris.topic2table.map": "topic_test:test_kafka_tbl", +"buffer.count.records":"1", +"buffer.flush.time":"120", +"buffer.size.bytes":"500", +"doris.urls":"10.10.10.1", +"doris.user":"root", +"doris.password":"", +"doris.http.port":"8030", +"doris.query.port":"9030", +"doris.database":"test_db", +"key.converter":"org.apache.kafka.connect.storage.StringConverter", +"value.converter":"org.apache.kafka.connect.json.JsonConverter", +"key.converter.schemas.enable":"false", +"value.converter.schemas.enable":"false", + } +}' +``` + +Operation Connector +``` +# View connector status +curl -i http://127.0.0.1:8083/connectors/test-doris-sink-cluster/status -X GET +# Delete connector +curl -i http://127.0.0.1:8083/connectors/test-doris-sink-cluster -X DELETE +# Pause connector +curl -i http://127.0.0.1:8083/connectors/test-doris-sink-cluster/pause -X PUT +# Restart connector +curl -i http://127.0.0.1:8083/connectors/test-doris-sink-cluster/resume -X PUT +# Restart tasks within the connector +curl -i http://127.0.0.1:8083/connectors/test-doris-sink-cluster/tasks/0/restart -X POST +``` +Refer to: [Connect REST Interface](https://docs.confluent.io/platform/current/connect/references/restapi.html#kconnect-rest-interface) + +:::warning +Note that when kafka-connect is started for the first time, three topics `config.storage.topic` `offset.storage.topic` and `status.storage.topic` will be created in the kafka cluster to record the shared connector configuration of ka
Re: [PR] [bugfix](coordinator) should use fragment id not profile fragment id to cancel fragment [doris]
yiguolei commented on PR #31852: URL: https://github.com/apache/doris/pull/31852#issuecomment-1980457179 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [type](doc)Standardize the name and modify the format [doris]
github-actions[bot] commented on PR #31636: URL: https://github.com/apache/doris/pull/31636#issuecomment-1980457682 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] (cloud-merge) Fix the serialization of CloudTablet [doris]
Lchangliang opened a new pull request, #31864: URL: https://github.com/apache/doris/pull/31864 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (cloud-merge) Fix the serialization of CloudTablet [doris]
doris-robot commented on PR #31864: URL: https://github.com/apache/doris/pull/31864#issuecomment-1980458036 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (cloud-merge) Fix the serialization of CloudTablet [doris]
Lchangliang commented on PR #31864: URL: https://github.com/apache/doris/pull/31864#issuecomment-1980458291 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature-WIP](partition) Suppor null range partition [doris]
doris-robot commented on PR #31827: URL: https://github.com/apache/doris/pull/31827#issuecomment-198041 TeamCity be ut coverage result: Function Coverage: 35.41% (8559/24174) Line Coverage: 27.16% (69486/255819) Region Coverage: 26.37% (36037/136639) Branch Coverage: 23.21% (18415/79342) Coverage Report: http://coverage.selectdb-in.cc/coverage/cfaf4663eea12e8bc9efe11149e685f91da78912_cfaf4663eea12e8bc9efe11149e685f91da78912/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](partition) fix npe when prune partition with not exist partition column in mv [doris]
BiteThet commented on PR #31860: URL: https://github.com/apache/doris/pull/31860#issuecomment-1980467836 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) Fix FE cannot work when restarting after schema change run in cloud mode [doris]
dataroaring merged PR #31842: URL: https://github.com/apache/doris/pull/31842 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](cloud) Fix FE cannot work when restarting after schema change (#31842)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 2f3d5c5e135 [fix](cloud) Fix FE cannot work when restarting after schema change (#31842) 2f3d5c5e135 is described below commit 2f3d5c5e135e1363801c38c9b0cc8c261ae58356 Author: deardeng <565620...@qq.com> AuthorDate: Wed Mar 6 17:45:28 2024 +0800 [fix](cloud) Fix FE cannot work when restarting after schema change (#31842) ``` 2024-03-05 21:16:43,300 INFO (stateListener|110) [RollupJobV2.replayPendingJob():749] replay waiting txn rollup job: 30405 2024-03-05 21:16:43,300 INFO (stateListener|110) [MaterializedViewHandler.replayAlterJobV2():1114] set table's state to ROLLUP, table id: 29056, job id: 30405 2024-03-05 21:16:43,300 DEBUG (stateListener|110) [Env.replayJournal():2751] journal 21241 replayed. 2024-03-05 21:16:43,300 DEBUG (stateListener|110) [JournalObservable.notifyObservers():95] notify observers: journal: 21241, pos: 0, size: 0, obs: [] 2024-03-05 21:16:43,300 DEBUG (stateListener|110) [JournalEntity.readFields():193] get opcode: 45 2024-03-05 21:16:43,300 DEBUG (stateListener|110) [EditLog.loadJournal():170] replay journal op code: 45 2024-03-05 21:16:43,301 DEBUG (stateListener|110) [InternalCatalog.unprotectUpdateReplica():1048] replay update a replica table id: 29056 partition id: 29050 index id: 30406 tablet id: 30505 backend id: 10002 replica id: 30506 version: 2 schema hash: -1 data size: 0 row count: 0 last failed version: -1 last success version: 2 2024-03-05 21:16:43,301 DEBUG (stateListener|110) [CloudReplica.getBackendId():143] connect context is null in getBackendId 2024-03-05 21:16:43,301 ERROR (stateListener|110) [EditLog.loadJournal():1226] Operation Type 45 java.lang.NullPointerException: table id: 29056 partition id: 29050 index id: 30406 tablet id: 30505 backend id: 10002 replica id: 30506 version: 2 schema hash: -1 data size: 0 row count: 0 last failed version: -1 last success version: 2 at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:921) ~[guava-32.1.2-jre.jar:?] at org.apache.doris.datasource.InternalCatalog.unprotectUpdateReplica(InternalCatalog.java:1054) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.InternalCatalog.replayUpdateReplica(InternalCatalog.java:1075) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.replayUpdateReplica(Env.java:3706) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.persist.EditLog.loadJournal(EditLog.java:398) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.replayJournal(Env.java:2747) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.transferToMaster(Env.java:1459) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.access$1400(Env.java:319) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env$5.runOneCycle(Env.java:2638) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT] ``` --- .../src/main/java/org/apache/doris/datasource/InternalCatalog.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java index 95abc9e06c4..3b689f44ba3 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java @@ -1050,7 +1050,7 @@ public class InternalCatalog implements CatalogIf { Partition partition = olapTable.getPartition(info.getPartitionId()); MaterializedIndex materializedIndex = partition.getIndex(info.getIndexId()); Tablet tablet = materializedIndex.getTablet(info.getTabletId()); -Replica replica = tablet.getReplicaByBackendId(info.getBackendId()); +Replica replica = tablet.getReplicaById(info.getReplicaId()); Preconditions.checkNotNull(replica, info); replica.updateVersionInfo(info.getVersion(), info.getDataSize(), info.getRemoteDataSize(), info.getRowCount()); replica.setBad(false); - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) fix create tablet miss cluster key index [doris]
yujun777 commented on PR #31862: URL: https://github.com/apache/doris/pull/31862#issuecomment-1980471637 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](io) Set segment file size to rowset meta [doris]
github-actions[bot] commented on PR #31853: URL: https://github.com/apache/doris/pull/31853#issuecomment-1980475093 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Nereids] add leading tpc-h [doris]
xzj7019 commented on code in PR #30405: URL: https://github.com/apache/doris/pull/30405#discussion_r1514157352 ## regression-test/data/nereids_hint_tpch_p0/shape/q10.out: ## @@ -2,33 +2,34 @@ -- !select -- PhysicalResultSink --PhysicalTopN[MERGE_SORT] -PhysicalDistribute +PhysicalDistribute[DistributionSpecGather] --PhysicalTopN[LOCAL_SORT] -hashAgg[GLOBAL] ---PhysicalDistribute -hashAgg[LOCAL] ---PhysicalProject -hashJoin[INNER_JOIN] hashCondition=((lineitem.l_orderkey = orders.o_orderkey)) otherCondition=() build RFs:RF2 o_orderkey->[l_orderkey] ---PhysicalProject -filter((lineitem.l_returnflag = 'R')) ---PhysicalOlapScan[lineitem] apply RFs: RF2 ---PhysicalDistribute +PhysicalProject +--hashAgg[GLOBAL] +PhysicalDistribute[DistributionSpecHash] +--hashAgg[LOCAL] +PhysicalProject +--hashJoin[INNER_JOIN] hashCondition=((lineitem.l_orderkey = orders.o_orderkey)) otherCondition=() PhysicalProject ---hashJoin[INNER_JOIN] hashCondition=((customer.c_nationkey = nation.n_nationkey)) otherCondition=() build RFs:RF1 n_nationkey->[c_nationkey] -PhysicalDistribute +--filter((lineitem.l_returnflag = 'R')) +PhysicalOlapScan[lineitem] +PhysicalDistribute[DistributionSpecHash] +--PhysicalProject +hashJoin[INNER_JOIN] hashCondition=((customer.c_nationkey = nation.n_nationkey)) otherCondition=() --PhysicalProject -hashJoin[INNER_JOIN] hashCondition=((customer.c_custkey = orders.o_custkey)) otherCondition=() build RFs:RF0 o_custkey->[c_custkey] +hashJoin[INNER_JOIN] hashCondition=((customer.c_custkey = orders.o_custkey)) otherCondition=() --PhysicalProject -PhysicalOlapScan[customer] apply RFs: RF0 RF1 ---PhysicalDistribute +PhysicalOlapScan[customer] +--PhysicalDistribute[DistributionSpecHash] PhysicalProject --filter((orders.o_orderdate < '1994-01-01') and (orders.o_orderdate >= '1993-10-01')) PhysicalOlapScan[orders] -PhysicalDistribute ---PhysicalProject -PhysicalOlapScan[nation] +--PhysicalDistribute[DistributionSpecReplicated] +PhysicalProject +--PhysicalOlapScan[nation] -Used: leading(lineitem { { customer orders } nation }) -UnUsed: +Hint log: +Used: leading(lineitem { { customer shuffle orders } broadcast nation } ) +UnUsed: [shuffle]_2 Review Comment: why this "UnUsed" exists? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [RuntimeFilter] (JDBC) jdbc scan node support IN/MIN_MAX runtime filter [doris]
doris-robot commented on PR #31741: URL: https://github.com/apache/doris/pull/31741#issuecomment-1980476395 TeamCity be ut coverage result: Function Coverage: 35.40% (8559/24179) Line Coverage: 27.14% (69481/256002) Region Coverage: 26.36% (36041/136716) Branch Coverage: 23.19% (18420/79428) Coverage Report: http://coverage.selectdb-in.cc/coverage/919e337c3737b6600be4a8eae6ef3da825ddcaf7_919e337c3737b6600be4a8eae6ef3da825ddcaf7/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Nereids] add leading tpc-h [doris]
xzj7019 commented on code in PR #30405: URL: https://github.com/apache/doris/pull/30405#discussion_r1514158605 ## regression-test/data/nereids_hint_tpch_p0/shape/q16.out: ## @@ -2,25 +2,26 @@ -- !select -- PhysicalResultSink --PhysicalQuickSort[MERGE_SORT] -PhysicalDistribute +PhysicalDistribute[DistributionSpecGather] --PhysicalQuickSort[LOCAL_SORT] hashAgg[GLOBAL] ---PhysicalDistribute +--PhysicalDistribute[DistributionSpecHash] hashAgg[LOCAL] ---hashJoin[LEFT_ANTI_JOIN] hashCondition=((partsupp.ps_suppkey = supplier.s_suppkey)) otherCondition=() -PhysicalProject ---hashJoin[INNER_JOIN] hashCondition=((part.p_partkey = partsupp.ps_partkey)) otherCondition=() build RFs:RF0 p_partkey->[ps_partkey] +--PhysicalProject +hashJoin[INNER_JOIN] hashCondition=((part.p_partkey = partsupp.ps_partkey)) otherCondition=() +--hashJoin[LEFT_ANTI_JOIN] hashCondition=((partsupp.ps_suppkey = supplier.s_suppkey)) otherCondition=() PhysicalProject ---PhysicalOlapScan[partsupp] apply RFs: RF0 -PhysicalProject ---filter(( not (p_brand = 'Brand#45')) and ( not (p_type like 'MEDIUM POLISHED%')) and p_size IN (14, 19, 23, 3, 36, 45, 49, 9)) -PhysicalOlapScan[part] -PhysicalDistribute +--PhysicalOlapScan[partsupp] +PhysicalDistribute[DistributionSpecReplicated] +--PhysicalProject +filter((s_comment like '%Customer%Complaints%')) +--PhysicalOlapScan[supplier] --PhysicalProject -filter((s_comment like '%Customer%Complaints%')) ---PhysicalOlapScan[supplier] +filter(( not (p_brand = 'Brand#45')) and ( not (p_type like 'MEDIUM POLISHED%')) and p_size IN (14, 19, 23, 3, 36, 45, 49, 9)) +--PhysicalOlapScan[part] -Used: leading(partsupp part supplier) -UnUsed: -SyntaxError: +Hint log: +Used: +UnUsed: [shuffle]_2 Review Comment: this UnUsed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Nereids] add leading tpc-h [doris]
xzj7019 commented on code in PR #30405: URL: https://github.com/apache/doris/pull/30405#discussion_r1514159381 ## regression-test/data/nereids_hint_tpch_p0/shape/q5.out: ## @@ -2,24 +2,43 @@ -- !select -- PhysicalResultSink --PhysicalQuickSort[MERGE_SORT] -PhysicalQuickSort[LOCAL_SORT] ---hashAgg[GLOBAL] -hashAgg[LOCAL] ---hashJoin[INNER_JOIN] hashCondition=((customer.c_custkey = orders.o_custkey) and (customer.c_nationkey = supplier.s_nationkey)) otherCondition=() build RFs:RF4 c_nationkey->[s_nationkey];RF5 c_custkey->[o_custkey] -hashJoin[INNER_JOIN] hashCondition=((lineitem.l_orderkey = orders.o_orderkey)) otherCondition=() build RFs:RF3 o_orderkey->[l_orderkey] ---hashJoin[INNER_JOIN] hashCondition=((lineitem.l_suppkey = supplier.s_suppkey)) otherCondition=() build RFs:RF2 s_suppkey->[l_suppkey] -PhysicalOlapScan[lineitem] apply RFs: RF2 RF3 -hashJoin[INNER_JOIN] hashCondition=((supplier.s_nationkey = nation.n_nationkey)) otherCondition=() build RFs:RF1 n_nationkey->[s_nationkey] ---PhysicalOlapScan[supplier] apply RFs: RF1 RF4 ---hashJoin[INNER_JOIN] hashCondition=((nation.n_regionkey = region.r_regionkey)) otherCondition=() build RFs:RF0 r_regionkey->[n_regionkey] -PhysicalOlapScan[nation] apply RFs: RF0 -filter((region.r_name = 'ASIA')) ---PhysicalOlapScan[region] ---filter((orders.o_orderdate < '1995-01-01') and (orders.o_orderdate >= '1994-01-01')) -PhysicalOlapScan[orders] apply RFs: RF5 -PhysicalOlapScan[customer] +PhysicalDistribute[DistributionSpecGather] +--PhysicalQuickSort[LOCAL_SORT] +hashAgg[GLOBAL] +--PhysicalDistribute[DistributionSpecHash] +hashAgg[LOCAL] +--PhysicalProject +hashJoin[INNER_JOIN] hashCondition=((customer.c_custkey = orders.o_custkey) and (customer.c_nationkey = supplier.s_nationkey)) otherCondition=() +--PhysicalDistribute[DistributionSpecHash] +PhysicalProject +--hashJoin[INNER_JOIN] hashCondition=((lineitem.l_suppkey = supplier.s_suppkey)) otherCondition=() +PhysicalProject +--hashJoin[INNER_JOIN] hashCondition=((lineitem.l_orderkey = orders.o_orderkey)) otherCondition=() +PhysicalProject +--PhysicalOlapScan[lineitem] +PhysicalProject +--filter((orders.o_orderdate < '1995-01-01') and (orders.o_orderdate >= '1994-01-01')) +PhysicalOlapScan[orders] +PhysicalDistribute[DistributionSpecReplicated] +--PhysicalProject +hashJoin[INNER_JOIN] hashCondition=((supplier.s_nationkey = nation.n_nationkey)) otherCondition=() +--PhysicalProject +PhysicalOlapScan[supplier] +--PhysicalDistribute[DistributionSpecReplicated] +PhysicalProject +--hashJoin[INNER_JOIN] hashCondition=((nation.n_regionkey = region.r_regionkey)) otherCondition=() +PhysicalProject +--PhysicalOlapScan[nation] +PhysicalDistribute[DistributionSpecReplicated] +--PhysicalProject +filter((region.r_name = 'ASIA')) +--PhysicalOlapScan[region] +--PhysicalDistribute[DistributionSpecHash] +PhysicalProject +--PhysicalOlapScan[customer] -Used: leading(lineitem { supplier { nation region } } orders customer) -UnUsed: +Hint log: +Used: leading(lineitem orders { supplier broadcast { nation broadcast region } } shuffle customer ) +UnUsed: [broadcast]_2 Review Comment: this one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Nereids] add leading tpc-h [doris]
xzj7019 commented on code in PR #30405: URL: https://github.com/apache/doris/pull/30405#discussion_r1514159943 ## regression-test/data/nereids_hint_tpch_p0/shape/q7.out: ## @@ -1,44 +1,44 @@ -- This file is automatically generated. You should know what you did if you want to edit this -- !select -- PhysicalResultSink ---PhysicalQuickSort -PhysicalDistribute ---PhysicalQuickSort +--PhysicalQuickSort[MERGE_SORT] +PhysicalDistribute[DistributionSpecGather] +--PhysicalQuickSort[LOCAL_SORT] hashAgg[GLOBAL] ---PhysicalDistribute +--PhysicalDistribute[DistributionSpecHash] hashAgg[LOCAL] --PhysicalProject -hashJoin[INNER_JOIN] hashCondition=((customer.c_custkey = orders.o_custkey) and (customer.c_nationkey = n2.n_nationkey))otherCondition=() +hashJoin[INNER_JOIN] hashCondition=((orders.o_orderkey = lineitem.l_orderkey) and (supplier.s_suppkey = lineitem.l_suppkey)) otherCondition=() --PhysicalProject -PhysicalOlapScan[customer] ---PhysicalDistribute -PhysicalProject ---hashJoin[INNER_JOIN] hashCondition=((orders.o_orderkey = lineitem.l_orderkey))otherCondition=() -PhysicalProject ---PhysicalOlapScan[orders] -PhysicalDistribute +filter((lineitem.l_shipdate <= '1996-12-31') and (lineitem.l_shipdate >= '1995-01-01')) +--PhysicalOlapScan[lineitem] +--PhysicalDistribute[DistributionSpecReplicated] +NestedLoopJoin[INNER_JOIN](((n1.n_name = 'FRANCE') AND (n2.n_name = 'GERMANY')) OR ((n1.n_name = 'GERMANY') AND (n2.n_name = 'FRANCE'))) +--PhysicalProject +hashJoin[INNER_JOIN] hashCondition=((supplier.s_nationkey = n1.n_nationkey)) otherCondition=() --PhysicalProject -hashJoin[INNER_JOIN] hashCondition=((supplier.s_suppkey = lineitem.l_suppkey))otherCondition=() ---PhysicalDistribute -hashJoin[INNER_JOIN] hashCondition=((supplier.s_nationkey = n1.n_nationkey))otherCondition=() ---PhysicalDistribute +PhysicalOlapScan[supplier] +--PhysicalDistribute[DistributionSpecReplicated] +PhysicalProject +--filter(n_name IN ('FRANCE', 'GERMANY')) +PhysicalOlapScan[nation] +--PhysicalDistribute[DistributionSpecReplicated] +PhysicalProject +--hashJoin[INNER_JOIN] hashCondition=((customer.c_custkey = orders.o_custkey)) otherCondition=() +PhysicalProject +--PhysicalOlapScan[orders] +PhysicalDistribute[DistributionSpecReplicated] +--PhysicalProject +hashJoin[INNER_JOIN] hashCondition=((customer.c_nationkey = n2.n_nationkey)) otherCondition=() +--PhysicalProject +PhysicalOlapScan[customer] +--PhysicalDistribute[DistributionSpecReplicated] PhysicalProject ---PhysicalOlapScan[supplier] ---PhysicalDistribute -NestedLoopJoin[INNER_JOIN](((n1.n_name = 'FRANCE') AND (n2.n_name = 'GERMANY')) OR ((n1.n_name = 'GERMANY') AND (n2.n_name = 'FRANCE'))) ---PhysicalProject -filter(((n2.n_name = 'GERMANY') OR (n2.n_name = 'FRANCE'))) ---PhysicalOlapScan[nation] ---PhysicalDistribute -PhysicalProject ---filter(((n1.n_name = 'FRANCE') OR (n1.n_name = 'GERMANY'))) -PhysicalOlapScan[nation] ---PhysicalDistribute -PhysicalProject ---filter((lineitem.l_shipdate <= '1996-12-31') and (lineitem.l_shipdate >= '1995-01-01')) -PhysicalOlapScan[lineitem] +--filter(n_name IN ('FRANCE', 'GERMANY')) +PhysicalOlapScan[nation] -Used: -UnUsed: -SyntaxError: leading(customer { orders { lineitem { supplier { n1 n2 } } } }) Msg:tables should be same as join tables +Hint log: +Used: leading(lineitem { supplier broadcast n1 } { orders { customer broadcast n2 } } )
Re: [PR] [Nereids] add leading tpc-h [doris]
xzj7019 commented on code in PR #30405: URL: https://github.com/apache/doris/pull/30405#discussion_r1514160750 ## regression-test/data/nereids_hint_tpch_p0/shape/q9.out: ## @@ -2,44 +2,45 @@ -- !select -- PhysicalResultSink --PhysicalQuickSort[MERGE_SORT] -PhysicalDistribute +PhysicalDistribute[DistributionSpecGather] --PhysicalQuickSort[LOCAL_SORT] hashAgg[GLOBAL] ---PhysicalDistribute +--PhysicalDistribute[DistributionSpecHash] hashAgg[LOCAL] --PhysicalProject -hashJoin[INNER_JOIN] hashCondition=((partsupp.ps_partkey = lineitem.l_partkey) and (partsupp.ps_suppkey = lineitem.l_suppkey)) otherCondition=() build RFs:RF4 ps_partkey->[l_partkey];RF5 ps_suppkey->[l_suppkey] ---PhysicalDistribute +hashJoin[INNER_JOIN] hashCondition=((partsupp.ps_partkey = lineitem.l_partkey) and (partsupp.ps_suppkey = lineitem.l_suppkey)) otherCondition=() +--PhysicalDistribute[DistributionSpecHash] PhysicalProject ---hashJoin[INNER_JOIN] hashCondition=((orders.o_orderkey = lineitem.l_orderkey)) otherCondition=() build RFs:RF3 l_orderkey->[o_orderkey] +--hashJoin[INNER_JOIN] hashCondition=((orders.o_orderkey = lineitem.l_orderkey)) otherCondition=() PhysicalProject ---PhysicalOlapScan[orders] apply RFs: RF3 -PhysicalDistribute +--PhysicalOlapScan[orders] +PhysicalDistribute[DistributionSpecHash] --PhysicalProject -hashJoin[INNER_JOIN] hashCondition=((supplier.s_suppkey = lineitem.l_suppkey)) otherCondition=() build RFs:RF2 s_suppkey->[l_suppkey] ---PhysicalDistribute -PhysicalProject ---hashJoin[INNER_JOIN] hashCondition=((part.p_partkey = lineitem.l_partkey)) otherCondition=() build RFs:RF1 p_partkey->[l_partkey] +hashJoin[INNER_JOIN] hashCondition=((supplier.s_nationkey = nation.n_nationkey)) otherCondition=() +--PhysicalProject +hashJoin[INNER_JOIN] hashCondition=((supplier.s_suppkey = lineitem.l_suppkey)) otherCondition=() +--PhysicalDistribute[DistributionSpecHash] +PhysicalProject +--hashJoin[INNER_JOIN] hashCondition=((part.p_partkey = lineitem.l_partkey)) otherCondition=() +PhysicalDistribute[DistributionSpecHash] +--PhysicalProject +PhysicalOlapScan[lineitem] +PhysicalDistribute[DistributionSpecHash] +--PhysicalProject +filter((p_name like '%green%')) +--PhysicalOlapScan[part] +--PhysicalDistribute[DistributionSpecHash] PhysicalProject ---PhysicalOlapScan[lineitem] apply RFs: RF1 RF2 RF4 RF5 -PhysicalDistribute ---PhysicalProject -filter((p_name like '%green%')) ---PhysicalOlapScan[part] ---PhysicalDistribute +--PhysicalOlapScan[supplier] +--PhysicalDistribute[DistributionSpecReplicated] PhysicalProject ---hashJoin[INNER_JOIN] hashCondition=((supplier.s_nationkey = nation.n_nationkey)) otherCondition=() build RFs:RF0 n_nationkey->[s_nationkey] -PhysicalDistribute ---PhysicalProject -PhysicalOlapScan[supplier] apply RFs: RF0 -PhysicalDistribute ---PhysicalProject -PhysicalOlapScan[nation] ---PhysicalDistribute +--PhysicalOlapScan[nation] +--PhysicalDistribute[DistributionSpecHash] PhysicalProject --PhysicalOlapScan[partsupp] -Used: leading(orders { { lineitem part } { supplier nation } } partsupp) -UnUsed: +Hint log: +Used: leading(orders { lineitem shuffle part } shuffle { supplier broadcast nation } shuffle partsupp ) +UnUsed: [shuffle]_2 Review Comment: this one -- This is an automated message from the Apache Git S
Re: [PR] [thirdparty](lib) Add base64 thirdparty lib [doris]
HappenLee merged PR #31850: URL: https://github.com/apache/doris/pull/31850 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (2f3d5c5e135 -> bd9a74ac9de)
This is an automated email from the ASF dual-hosted git repository. lihaopeng pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 2f3d5c5e135 [fix](cloud) Fix FE cannot work when restarting after schema change (#31842) add bd9a74ac9de [thirdparty](lib) Add base64 thirdparty lib (#31850) No new revisions were added by this update. Summary of changes: thirdparty/build-thirdparty.sh | 21 + thirdparty/vars.sh | 7 +++ 2 files changed, 28 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Fix](Nereids) fix hint cases with random result [doris]
LiBinfeng-01 opened a new pull request, #31865: URL: https://github.com/apache/doris/pull/31865 ## Proposed changes - remove explain shape plan of unused leading cases - use leading to fasten unstable test distribute case ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Nereids) fix hint cases with random result [doris]
doris-robot commented on PR #31865: URL: https://github.com/apache/doris/pull/31865#issuecomment-1980480105 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](http stream) http stream support memtable_on_sink_node header [doris]
mymeiyi opened a new pull request, #31866: URL: https://github.com/apache/doris/pull/31866 ## Proposed changes Make `memtable_on_sink_node` header work for `http_stream` ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](http stream) http stream support memtable_on_sink_node header [doris]
doris-robot commented on PR #31866: URL: https://github.com/apache/doris/pull/31866#issuecomment-1980485425 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [opt](session variable) max_msg_size_of_result_receiver #31809 [doris]
zhiqiang- opened a new pull request, #31867: URL: https://github.com/apache/doris/pull/31867 pick #31809 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](session variable) max_msg_size_of_result_receiver #31809 [doris]
zhiqiang- commented on PR #31867: URL: https://github.com/apache/doris/pull/31867#issuecomment-1980491089 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](session variable) max_msg_size_of_result_receiver #31809 [doris]
doris-robot commented on PR #31867: URL: https://github.com/apache/doris/pull/31867#issuecomment-1980490886 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](http stream) http stream support memtable_on_sink_node header [doris]
mymeiyi commented on PR #31866: URL: https://github.com/apache/doris/pull/31866#issuecomment-1980492493 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) fix create tablet miss cluster key index [doris]
yujun777 commented on PR #31862: URL: https://github.com/apache/doris/pull/31862#issuecomment-1980495141 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](struct)Enclose the value area in quotation marks when struct to… [doris]
github-actions[bot] commented on PR #30504: URL: https://github.com/apache/doris/pull/30504#issuecomment-1980496216 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[I] [Bug] (JDBC) Error while querying JDBC catalog [doris]
DarvenDuan opened a new issue, #31868: URL: https://github.com/apache/doris/issues/31868 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? Creating a JDBC catalog for querying mysql,and I got err msg bellow: `ERROR 1105 (HY000): errCode = 2, detailMessage = (11.91.162.209)[INTERNAL_ERROR]UdfRuntimeException: jdbc get block address: CAUSED BY: SQLException: Conversion not supported for type java.lang.Byte` mysql table: `CREATE TABLE mysql_test_tbl ( c_tinyint tinyint(4) DEFAULT NULL, c_smallint smallint(6) DEFAULT NULL, c_int int(11) DEFAULT NULL, c_bigint bigint(20) DEFAULT NULL, c_float float DEFAULT NULL, c_double double DEFAULT NULL, c_decimal decimal(10,0) DEFAULT NULL, c_char char(1) DEFAULT NULL, c_varchar varchar(64) DEFAULT NULL, c_text text, c_date date DEFAULT NULL, c_datetime datetime DEFAULT NULL, c_timestamp timestamp NULL DEFAULT NULL ) ENGINE=InnoDB` ### What You Expected? query succeed ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [type](doc)Standardize the name and modify the format [doris]
zy-kkk commented on PR #31636: URL: https://github.com/apache/doris/pull/31636#issuecomment-1980498162 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](cloud) fix create tablet miss cluster key index [doris]
yujun777 commented on PR #31862: URL: https://github.com/apache/doris/pull/31862#issuecomment-1980501420 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (cloud-merge) Fix the serialization of CloudTablet [doris]
github-actions[bot] commented on PR #31864: URL: https://github.com/apache/doris/pull/31864#issuecomment-1980505398 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (cloud-merge) Fix the serialization of CloudTablet [doris]
github-actions[bot] commented on PR #31864: URL: https://github.com/apache/doris/pull/31864#issuecomment-1980505488 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](http stream) http stream support memtable_on_sink_node header [doris]
github-actions[bot] commented on PR #31866: URL: https://github.com/apache/doris/pull/31866#issuecomment-1980510064 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](http stream) http stream support memtable_on_sink_node header [doris]
github-actions[bot] commented on PR #31866: URL: https://github.com/apache/doris/pull/31866#issuecomment-1980510146 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] Release Note 2.1.0 [doris]
Hanchers commented on issue #31770: URL: https://github.com/apache/doris/issues/31770#issuecomment-1980512886 great -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Fix](Nereids) fix missing comment when creating table [doris]
zy-kkk opened a new pull request, #31869: URL: https://github.com/apache/doris/pull/31869 ## Proposed changes Issue Number: close #31829 ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Nereids) fix missing comment when creating table [doris]
doris-robot commented on PR #31869: URL: https://github.com/apache/doris/pull/31869#issuecomment-1980515719 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org