[GitHub] [incubator-doris] morningman opened a new issue #3640: [Bug] BE restart failed because of reading tablet in DELETE status

morningman opened a new issue #3640: URL: https://github.com/apache/incubator-doris/issues/3640 **Describe the bug** Previously, we introduced a new BE parameter(#3632 ). When a tablet fails to load, BE startup fails. We found that sometimes when the BE starts, it reads the tablet

[GitHub] [incubator-doris] morningman opened a new issue #3639: [MetaTool] Support batch delete meta using meta_tool

morningman opened a new issue #3639: URL: https://github.com/apache/incubator-doris/issues/3639 Currently, `meta_tool` only support delete meta for a single tablet. If there are thousands of tablets need to be deleted, this tool run very slow because it will open the Rocksdb for each

[GitHub] [incubator-doris] yangzhg opened a new pull request #3638: Supoort utf-8 encoding in instr, locate, locate_pos, lpad, rpad

yangzhg opened a new pull request #3638: URL: https://github.com/apache/incubator-doris/pull/3638 Supoort utf-8 encoding for string function `instr`, `locate`, `locate_pos`, `lpad`, `rpad` and add unit test for them This

[GitHub] [incubator-doris] decster opened a new pull request #3637: [Memory Engine] Add MemSubTablet, MemTablet, WriteTx, PartialRowBatch

decster opened a new pull request #3637: URL: https://github.com/apache/incubator-doris/pull/3637 This CL adds MemSubTablet, MemTablet, WriteTx, PartialRowBatch classes, with partial write support. Part of #3466 T

[GitHub] [incubator-doris] kangkaisen commented on pull request #3601: fix large string val allocation failure, #3600

kangkaisen commented on pull request #3601: URL: https://github.com/apache/incubator-doris/pull/3601#issuecomment-631209330 > > > Large bitmap will need use StringVal to allocate large memory, which is large than MAX_INT. > > > The overflow will cause serialization failure of bitmap.

[GitHub] [incubator-doris] worker24h commented on a change in pull request #3553: Support load json-data into Doris by RoutineLoad or StreamLoad

worker24h commented on a change in pull request #3553: URL: https://github.com/apache/incubator-doris/pull/3553#discussion_r427716933 ## File path: be/test/exec/json_scanner_test.cpp ## @@ -0,0 +1,445 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

[GitHub] [incubator-doris] worker24h commented on a change in pull request #3553: Support load json-data into Doris by RoutineLoad or StreamLoad

worker24h commented on a change in pull request #3553: URL: https://github.com/apache/incubator-doris/pull/3553#discussion_r427717067 ## File path: be/test/exec/json_scanner_test2.cpp ## @@ -0,0 +1,426 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or mor

[GitHub] [incubator-doris] chaoyli merged pull request #3580: [Memory Engine] Add column reader/writer

chaoyli merged pull request #3580: URL: https://github.com/apache/incubator-doris/pull/3580 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[incubator-doris] branch master updated (6be7a62 -> c54cb4b)

This is an automated email from the ASF dual-hosted git repository. lichaoyong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 6be7a62 [Config] Add ignore config to determine whether to continue to start be when load tablet from

[GitHub] [incubator-doris] chaoyli closed issue #3465: [Memory Engine] Core column data structure

chaoyli closed issue #3465: URL: https://github.com/apache/incubator-doris/issues/3465 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3571: Support bitmap_intersect

kangkaisen commented on a change in pull request #3571: URL: https://github.com/apache/incubator-doris/pull/3571#discussion_r427714328 ## File path: be/src/exprs/bitmap_function.cpp ## @@ -302,6 +302,31 @@ void BitmapFunctions::bitmap_union(FunctionContext* ctx, const StringVa

[GitHub] [incubator-doris] kangkaisen commented on pull request #3571: Support bitmap_intersect

kangkaisen commented on pull request #3571: URL: https://github.com/apache/incubator-doris/pull/3571#issuecomment-631204907 @EmmyMiao87 Hi. 当前对于 bitmap_intersect null的处理是不合理的。 按照SQL标准，sum(null), min(null), max(null)结果为 null. count(null)为0. 所以 对于 bitmap_intersect，bitmap_uni

[GitHub] [incubator-doris] kangpinghuang commented on a change in pull request #3601: fix large string val allocation failure, #3600

kangpinghuang commented on a change in pull request #3601: URL: https://github.com/apache/incubator-doris/pull/3601#discussion_r427707616 ## File path: be/src/exprs/bitmap_function.cpp ## @@ -458,7 +458,8 @@ StringVal BitmapFunctions::bitmap_from_string(FunctionContext* ctx, c

[GitHub] [incubator-doris] kangpinghuang commented on pull request #3601: fix large string val allocation failure, #3600

kangpinghuang commented on pull request #3601: URL: https://github.com/apache/incubator-doris/pull/3601#issuecomment-631196531 > > Large bitmap will need use StringVal to allocate large memory, which is large than MAX_INT. > > The overflow will cause serialization failure of bitmap. >

[GitHub] [incubator-doris] kangpinghuang removed a comment on pull request #3601: fix large string val allocation failure, #3600

kangpinghuang removed a comment on pull request #3601: URL: https://github.com/apache/incubator-doris/pull/3601#issuecomment-631196411 > than yes, There is a user who use the bitmap to do the count distinct. In his situation, the dimension is about 30billion, the bitmap I see is abou

[GitHub] [incubator-doris] kangpinghuang commented on pull request #3601: fix large string val allocation failure, #3600

kangpinghuang commented on pull request #3601: URL: https://github.com/apache/incubator-doris/pull/3601#issuecomment-631196411 > than yes, There is a user who use the bitmap to do the count distinct. In his situation, the dimension is about 30billion, the bitmap I see is about 3G.

[GitHub] [incubator-doris] vagetablechicken commented on issue #3468: [Memory Engine] Write pipeline external API and data format

vagetablechicken commented on issue #3468: URL: https://github.com/apache/incubator-doris/issues/3468#issuecomment-631195933 > Are all numeric values using string? Is it supported to use json number? I've forgot to provide one such example. Ref https://github.com/apache/incubator-do

[GitHub] [incubator-doris] kangkaisen commented on pull request #3626: Update Compiling environment of BE to Support C++14/17

kangkaisen commented on pull request #3626: URL: https://github.com/apache/incubator-doris/pull/3626#issuecomment-631190697 @HappenLee Hi，You should also update this http://doris.apache.org/master/zh-CN/installing/compilation.html#%E7%9B%B4%E6%8E%A5%E7%BC%96%E8%AF%91%EF%BC%88centos-ubuntu%

[GitHub] [incubator-doris] morningman merged pull request #3632: Add ignore config to determine whether to continue to start be when load tablet from header failed.

morningman merged pull request #3632: URL: https://github.com/apache/incubator-doris/pull/3632 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[incubator-doris] branch master updated: [Config] Add ignore config to determine whether to continue to start be when load tablet from header failed. (#3632)

This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new 6be7a62 [Config] Add ignore config

[GitHub] [incubator-doris] morningman closed issue #3633: [Config] Stop BE start process when loading tablet header failed

morningman closed issue #3633: URL: https://github.com/apache/incubator-doris/issues/3633 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [incubator-doris] morningman closed issue #3627: Doris be cannot start after upgrade to 0.12.10

morningman closed issue #3627: URL: https://github.com/apache/incubator-doris/issues/3627 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[incubator-doris] branch master updated: [Bug] Fix first start error after upgrade doris to support delete dulplicate table value columns (#3628)

This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new 58a6628 [Bug] Fix first start erro

[GitHub] [incubator-doris] morningman merged pull request #3628: Fix first start error after upgrade doris to support delete dulplicate table value columns

morningman merged pull request #3628: URL: https://github.com/apache/incubator-doris/pull/3628 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [incubator-doris] kangkaisen commented on pull request #3601: fix large string val allocation failure, #3600

kangkaisen commented on pull request #3601: URL: https://github.com/apache/incubator-doris/pull/3601#issuecomment-630943474 > Large bitmap will need use StringVal to allocate large memory, which is large than MAX_INT. > The overflow will cause serialization failure of bitmap. @kan

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3604: (#3464) fix Query failed when fact table has no data in join case

kangkaisen commented on a change in pull request #3604: URL: https://github.com/apache/incubator-doris/pull/3604#discussion_r42736 ## File path: fe/src/main/java/org/apache/doris/qe/Coordinator.java ## @@ -914,17 +914,29 @@ private void computeFragmentHosts() throws Excepti

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3604: (#3464) fix Query failed when fact table has no data in join case

kangkaisen commented on a change in pull request #3604: URL: https://github.com/apache/incubator-doris/pull/3604#discussion_r427441551 ## File path: fe/src/main/java/org/apache/doris/qe/Coordinator.java ## @@ -914,17 +914,29 @@ private void computeFragmentHosts() throws Excepti

[GitHub] [incubator-doris] kangkaisen commented on issue #3624: [Proposal] The execution model of compaction needs to improve

kangkaisen commented on issue #3624: URL: https://github.com/apache/incubator-doris/issues/3624#issuecomment-630923588 > Good proposal. > > However, the compaction strategy is difficult to meet all needs. In the case of inaccurate memory statistics, manual intervention may still be

[GitHub] [incubator-doris] imay commented on a change in pull request #3571: Support bitmap_intersect

imay commented on a change in pull request #3571: URL: https://github.com/apache/incubator-doris/pull/3571#discussion_r427413916 ## File path: docs/en/sql-reference/sql-functions/bitmap-functions/bitmap_intersect.md ## @@ -0,0 +1,54 @@ +--- +{ +"title": "bitmap_intersect",

[GitHub] [incubator-doris] HappenLee commented on a change in pull request #3626: Update Compiling environment of BE to Support C++14/17

HappenLee commented on a change in pull request #3626: URL: https://github.com/apache/incubator-doris/pull/3626#discussion_r427414647 ## File path: env.sh ## @@ -55,12 +55,19 @@ if [[ -z ${DORIS_GCC_HOME} ]]; then fi gcc_ver=`${DORIS_GCC_HOME}/bin/gcc -dumpfullversion -dump

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3571: Support bitmap_intersect

kangkaisen commented on a change in pull request #3571: URL: https://github.com/apache/incubator-doris/pull/3571#discussion_r427403092 ## File path: be/src/exprs/bitmap_function.cpp ## @@ -302,6 +302,31 @@ void BitmapFunctions::bitmap_union(FunctionContext* ctx, const StringVa

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3553: Support load json-data into Doris by RoutineLoad or StreamLoad

kangkaisen commented on a change in pull request #3553: URL: https://github.com/apache/incubator-doris/pull/3553#discussion_r427397193 ## File path: be/test/exec/json_scanner_test.cpp ## @@ -0,0 +1,445 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or mor

[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3553: Support load json-data into Doris by RoutineLoad or StreamLoad

kangkaisen commented on a change in pull request #3553: URL: https://github.com/apache/incubator-doris/pull/3553#discussion_r427396643 ## File path: be/test/exec/json_scanner_test2.cpp ## @@ -0,0 +1,426 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or mo

[GitHub] [incubator-doris] morningman commented on a change in pull request #3418: [Spark load] Add spark etl cluster and cluster manager

morningman commented on a change in pull request #3418: URL: https://github.com/apache/incubator-doris/pull/3418#discussion_r427356073 ## File path: docs/zh-CN/sql-reference/sql-statements/Account Management/GRANT.md ## @@ -33,6 +33,8 @@ Syntax: GRANT privilege_list ON d

[GitHub] [incubator-doris] imay commented on a change in pull request #3626: Update Compiling environment of BE to Support C++14/17

imay commented on a change in pull request #3626: URL: https://github.com/apache/incubator-doris/pull/3626#discussion_r427371806 ## File path: env.sh ## @@ -55,12 +55,19 @@ if [[ -z ${DORIS_GCC_HOME} ]]; then fi gcc_ver=`${DORIS_GCC_HOME}/bin/gcc -dumpfullversion -dumpversi

[GitHub] [incubator-doris] imay commented on a change in pull request #3626: Update Compiling environment of BE to Support C++14/17

imay commented on a change in pull request #3626: URL: https://github.com/apache/incubator-doris/pull/3626#discussion_r427371183 ## File path: be/CMakeLists.txt ## @@ -264,7 +264,7 @@ check_function_exists(sched_getcpu HAVE_SCHED_GETCPU) # -fno-omit-frame-pointers: Keep frame

[GitHub] [incubator-doris] wyb commented on a change in pull request #3418: [Spark load] Add spark etl cluster and cluster manager

wyb commented on a change in pull request #3418: URL: https://github.com/apache/incubator-doris/pull/3418#discussion_r427361798 ## File path: docs/zh-CN/administrator-guide/load-data/spark-load-manual.md ## @@ -0,0 +1,351 @@ +---

[GitHub] [incubator-doris] wyb commented on a change in pull request #3418: [Spark load] Add spark etl cluster and cluster manager

wyb commented on a change in pull request #3418: URL: https://github.com/apache/incubator-doris/pull/3418#discussion_r427358330 ## File path: fe/src/main/java/org/apache/doris/analysis/CreateResourceStmt.java ## @@ -0,0 +1,100 @@ +// Licensed to the Apache Software Foundation (

[GitHub] [incubator-doris] morningman commented on a change in pull request #3584: [OUTFILE] Support `INTO OUTFILE` to export query result

morningman commented on a change in pull request #3584: URL: https://github.com/apache/incubator-doris/pull/3584#discussion_r427307837 ## File path: docs/zh-CN/administrator-guide/outfile.md ## @@ -0,0 +1,183 @@ +--- +{ +"title": "导出查询结果集", +"language": "zh-CN" +} +---

[GitHub] [incubator-doris] morningman commented on pull request #3635: [Enhancement] Convert metric to Json format

morningman commented on pull request #3635: URL: https://github.com/apache/incubator-doris/pull/3635#issuecomment-630823108 Why we need a json format metric output? This is an automated message from the Apache Git Service. To

[GitHub] [incubator-doris] morningman commented on a change in pull request #3635: [Enhancement] Convert metric to Json format

morningman commented on a change in pull request #3635: URL: https://github.com/apache/incubator-doris/pull/3635#discussion_r427293584 ## File path: be/src/http/action/metrics_action.cpp ## @@ -143,15 +143,74 @@ void SimpleCoreMetricsVisitor::visit(const std::string& prefix,

[GitHub] [incubator-doris] decster commented on issue #3468: [Memory Engine] Write pipeline external API and data format

decster commented on issue #3468: URL: https://github.com/apache/incubator-doris/issues/3468#issuecomment-630802783 Are all numeric values using string? Is it supported to use json number? This is an automated message from

[GitHub] [incubator-doris] chaoyli commented on pull request #3635: [Enhancement] Convert metric to Json format

chaoyli commented on pull request #3635: URL: https://github.com/apache/incubator-doris/pull/3635#issuecomment-630784998 #3636 This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [incubator-doris] chaoyli opened a new issue #3636: Add a JSON format for existing metrics like this

chaoyli opened a new issue #3636: URL: https://github.com/apache/incubator-doris/issues/3636 The format is like this. ``` { "tags": { "metric":"thread_pool", "name":"thrift-server-pool", "type":"active_thread_num" }, "unit":"

[GitHub] [incubator-doris] chaoyli opened a new pull request #3635: [Feature] Convert metric to Json format

chaoyli opened a new pull request #3635: URL: https://github.com/apache/incubator-doris/pull/3635 Add a JSON format for existing metrics like this. ``` { "tags": { "metric":"thread_pool", "name":"thrift-server-pool", "type":"active_thread_

[GitHub] [incubator-doris] HappenLee commented on a change in pull request #3626: Update Compiling environment of BE to Support C++14/17

HappenLee commented on a change in pull request #3626: URL: https://github.com/apache/incubator-doris/pull/3626#discussion_r427255522 ## File path: be/CMakeLists.txt ## @@ -264,7 +264,7 @@ check_function_exists(sched_getcpu HAVE_SCHED_GETCPU) # -fno-omit-frame-pointers: Keep

[GitHub] [incubator-doris] HappenLee commented on a change in pull request #3626: Update Compiling environment of BE to Support C++14/17

HappenLee commented on a change in pull request #3626: URL: https://github.com/apache/incubator-doris/pull/3626#discussion_r427255522 ## File path: be/CMakeLists.txt ## @@ -264,7 +264,7 @@ check_function_exists(sched_getcpu HAVE_SCHED_GETCPU) # -fno-omit-frame-pointers: Keep

[GitHub] [incubator-doris] wutiangan commented on a change in pull request #3632: Add ignore config to determine whether to continue to start be when load tablet from header failed.

wutiangan commented on a change in pull request #3632: URL: https://github.com/apache/incubator-doris/pull/3632#discussion_r427247972 ## File path: docs/zh-CN/administrator-guide/config/be_config.md ## @@ -396,3 +396,8 @@ under the License. ### `webserver_port` ### `write_b

[GitHub] [incubator-doris] HappenLee commented on a change in pull request #3626: Update Compiling environment of BE to Support C++14/17

HappenLee commented on a change in pull request #3626: URL: https://github.com/apache/incubator-doris/pull/3626#discussion_r427247372 ## File path: env.sh ## @@ -55,12 +55,19 @@ if [[ -z ${DORIS_GCC_HOME} ]]; then fi gcc_ver=`${DORIS_GCC_HOME}/bin/gcc -dumpfullversion -dump

[incubator-doris] branch master updated: [Bug] instance mem tracker should has no limit (#3592)

This is an automated email from the ASF dual-hosted git repository. kangkaisen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new 9425f17 [Bug] instance mem tracker

[GitHub] [incubator-doris] kangkaisen closed pull request #3531: Remove unreasonable DCHECK for mem_tracker

kangkaisen closed pull request #3531: URL: https://github.com/apache/incubator-doris/pull/3531 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [incubator-doris] kangkaisen closed issue #3591: BE cores inside ReservationTracker in debug mode

kangkaisen closed issue #3591: URL: https://github.com/apache/incubator-doris/issues/3591 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [incubator-doris] kangkaisen merged pull request #3592: [Bug] instance mem tracker should has no limit

kangkaisen merged pull request #3592: URL: https://github.com/apache/incubator-doris/pull/3592 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [incubator-doris] sebbASF closed issue #3567: Download page issues

sebbASF closed issue #3567: URL: https://github.com/apache/incubator-doris/issues/3567 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [incubator-doris] sebbASF commented on issue #3567: Download page issues

sebbASF commented on issue #3567: URL: https://github.com/apache/incubator-doris/issues/3567#issuecomment-630745608 Looks OK now, thanks This is an automated message from the Apache Git Service. To respond to the message, pl

[GitHub] [incubator-doris] sebbASF commented on issue #3568: Do not link to dist.apache.org from public pages

sebbASF commented on issue #3568: URL: https://github.com/apache/incubator-doris/issues/3568#issuecomment-630745113 The reference to dist.a.o has been removed, thanks. However, I have just noticed that the verification page suggests using RAT to verify the source licenses. The RAT t

[GitHub] [incubator-doris] imay commented on a change in pull request #3626: Update Compiling environment of BE to Support C++14/17

imay commented on a change in pull request #3626: URL: https://github.com/apache/incubator-doris/pull/3626#discussion_r427170693 ## File path: be/CMakeLists.txt ## @@ -264,7 +264,7 @@ check_function_exists(sched_getcpu HAVE_SCHED_GETCPU) # -fno-omit-frame-pointers: Keep frame

[GitHub] [incubator-doris] ZhangHuirui commented on issue #3543: Doris ROUTINE LOAD使用kafka做数据源，导入是否保证分区有序？

ZhangHuirui commented on issue #3543: URL: https://github.com/apache/incubator-doris/issues/3543#issuecomment-630713658 I see. Ordered Message is Not Supported? This is an automated message from the Apache Git Service. T

[incubator-doris] branch master updated (4cbcae1 -> 8018b1c)

This is an automated email from the ASF dual-hosted git repository. zhaoc pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 4cbcae1 [Spark on Doris] Shade and provide the thrift lib in spark-doris-connector (#3631) add 8018b1c

[GitHub] [incubator-doris] imay merged pull request #3602: [Doris On ES]fix bug of like not translate correcttly

imay merged pull request #3602: URL: https://github.com/apache/incubator-doris/pull/3602 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[GitHub] [incubator-doris] imay commented on a change in pull request #3580: [Memory Engine] Add column reader/writer

imay commented on a change in pull request #3580: URL: https://github.com/apache/incubator-doris/pull/3580#discussion_r427146376 ## File path: be/src/olap/memory/typed_column_writer.h ## @@ -0,0 +1,305 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or mor

[GitHub] [incubator-doris] worker24h commented on a change in pull request #3553: Support load json-data into Doris by RoutineLoad or StreamLoad

worker24h commented on a change in pull request #3553: URL: https://github.com/apache/incubator-doris/pull/3553#discussion_r427115410 ## File path: docs/zh-CN/sql-reference/sql-statements/Data Manipulation/STREAM LOAD.md ## @@ -67,13 +67,28 @@ under the License. 比如指定导

[GitHub] [incubator-doris] worker24h commented on a change in pull request #3553: Support load json-data into Doris by RoutineLoad or StreamLoad

worker24h commented on a change in pull request #3553: URL: https://github.com/apache/incubator-doris/pull/3553#discussion_r427115347 ## File path: docs/zh-CN/sql-reference/sql-statements/Data Manipulation/STREAM LOAD.md ## @@ -127,7 +142,35 @@ under the License. 9. 导入含有B

[GitHub] [incubator-doris] worker24h commented on a change in pull request #3553: Support load json-data into Doris by RoutineLoad or StreamLoad

worker24h commented on a change in pull request #3553: URL: https://github.com/apache/incubator-doris/pull/3553#discussion_r427114669 ## File path: docs/en/sql-reference/sql-statements/Data Manipulation/ROUTINE LOAD.md ## @@ -163,10 +163,21 @@ FROM data_source Wheth