kangkaisen commented on issue #1733: Free olap scanner out of lock
URL: https://github.com/apache/incubator-doris/pull/1733#issuecomment-527017035
> > @imay Hi, do you have a benchmark test for this PR?
>
> @kangkaisen
>
> This only one reason leading to competition.
> After
imay commented on issue #1732: Add exchange in MemPool to reduce alloc/free
operation
URL: https://github.com/apache/incubator-doris/pull/1732#issuecomment-527017158
> This PR seems have a indent issue?
Origin code is 2 spaces indent, I change it to 4 spaces which is consistent
with
kangkaisen commented on a change in pull request #1732: Add exchange in MemPool
to reduce alloc/free operation
URL: https://github.com/apache/incubator-doris/pull/1732#discussion_r319821901
##
File path: be/src/runtime/mem_pool.cpp
##
@@ -26,248 +26,239 @@
#include "com
kangkaisen commented on issue #1733: Free olap scanner out of lock
URL: https://github.com/apache/incubator-doris/pull/1733#issuecomment-527021936
+1. LGTM.
This is an automated message from the Apache Git Service.
To respond
imay commented on a change in pull request #1732: Add exchange in MemPool to
reduce alloc/free operation
URL: https://github.com/apache/incubator-doris/pull/1732#discussion_r319822402
##
File path: be/src/runtime/mem_pool.cpp
##
@@ -26,248 +26,239 @@
#include "common/na
kangkaisen commented on issue #1732: Add exchange in MemPool to reduce
alloc/free operation
URL: https://github.com/apache/incubator-doris/pull/1732#issuecomment-527023087
+1. LGTM.
The main change of this PR is add the `MemPool::exchange_data` method, other
diff is because of indent.
imay commented on a change in pull request #1678: Add rowset id generator to FE
and BE
URL: https://github.com/apache/incubator-doris/pull/1678#discussion_r319825362
##
File path: be/src/olap/rowset/segment_group.h
##
@@ -262,12 +262,12 @@ class SegmentGroup {
OLAPS
imay commented on a change in pull request #1678: Add rowset id generator to FE
and BE
URL: https://github.com/apache/incubator-doris/pull/1678#discussion_r319825420
##
File path: be/src/olap/rowset/segment_group.h
##
@@ -248,7 +248,7 @@ class SegmentGroup {
int64_t
imay commented on a change in pull request #1678: Add rowset id generator to FE
and BE
URL: https://github.com/apache/incubator-doris/pull/1678#discussion_r319824694
##
File path: be/src/olap/storage_engine.h
##
@@ -196,6 +197,12 @@ class StorageEngine {
// TODO(ygl)
kangpinghuang commented on a change in pull request #1718: Basic implementation
for BetaRowsetReader
URL: https://github.com/apache/incubator-doris/pull/1718#discussion_r319830181
##
File path: be/src/olap/rowset/beta_rowset_reader.cpp
##
@@ -0,0 +1,122 @@
+// Licensed to
gaodayue commented on a change in pull request #1718: Basic implementation for
BetaRowsetReader
URL: https://github.com/apache/incubator-doris/pull/1718#discussion_r319841547
##
File path: be/src/olap/rowset/beta_rowset_reader.cpp
##
@@ -0,0 +1,122 @@
+// Licensed to the A
gaodayue commented on a change in pull request #1718: Basic implementation for
BetaRowsetReader
URL: https://github.com/apache/incubator-doris/pull/1718#discussion_r319844813
##
File path: be/src/olap/rowset/beta_rowset_reader.cpp
##
@@ -0,0 +1,122 @@
+// Licensed to the A
imay merged pull request #1735: Fix NPE of DataDescription
URL: https://github.com/apache/incubator-doris/pull/1735
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
imay merged pull request #1733: Free olap scanner out of lock
URL: https://github.com/apache/incubator-doris/pull/1733
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitH
chaoyli merged pull request #1678: Add rowset id generator to FE and BE
URL: https://github.com/apache/incubator-doris/pull/1678
This is an automated message from the Apache Git Service.
To respond to the message, please log
imay merged pull request #1731: Add scroll keepalive and http timeout
configuration
URL: https://github.com/apache/incubator-doris/pull/1731
This is an automated message from the Apache Git Service.
To respond to the message
imay merged pull request #1732: Add exchange in MemPool to reduce alloc/free
operation
URL: https://github.com/apache/incubator-doris/pull/1732
This is an automated message from the Apache Git Service.
To respond to the mess
yiguolei opened a new pull request #1736: Move version to high 8 bit
URL: https://github.com/apache/incubator-doris/pull/1736
This is an automated message from the Apache Git Service.
To respond to the message, please log on
imay merged pull request #1736: Move version to high 8 bit
URL: https://github.com/apache/incubator-doris/pull/1736
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
EmmyMiao87 opened a new pull request #1737: Unify the msg of 'Memory exceed
limit'
URL: https://github.com/apache/incubator-doris/pull/1737
The new msg of limit exceed: "Memory exceed limit. %msg, Backend:%ip,
fragment:%id Used:% , Limit:%. xxx".
This commit unifies the msg of 'Memory e
kangpinghuang opened a new pull request #1738: add bloom filter index
URL: https://github.com/apache/incubator-doris/pull/1738
Add bloom filter index to prune data when loading segment
This is an automated message from the Apa
imay commented on a change in pull request #1737: Unify the msg of 'Memory
exceed limit'
URL: https://github.com/apache/incubator-doris/pull/1737#discussion_r319941741
##
File path: be/src/runtime/buffered_block_mgr.cpp
##
@@ -76,9 +77,9 @@ Status BufferedBlockMgr::get_new
imay commented on a change in pull request #1737: Unify the msg of 'Memory
exceed limit'
URL: https://github.com/apache/incubator-doris/pull/1737#discussion_r319941135
##
File path: be/src/exec/sort_node.cpp
##
@@ -146,8 +146,9 @@ Status SortNode::sort_input(RuntimeState*
imay commented on a change in pull request #1737: Unify the msg of 'Memory
exceed limit'
URL: https://github.com/apache/incubator-doris/pull/1737#discussion_r319941621
##
File path: be/src/runtime/plan_fragment_executor.cpp
##
@@ -514,7 +514,7 @@ void PlanFragmentExecutor:
imay commented on a change in pull request #1737: Unify the msg of 'Memory
exceed limit'
URL: https://github.com/apache/incubator-doris/pull/1737#discussion_r319941218
##
File path: be/src/exec/partitioned_aggregation_node.cc
##
@@ -232,8 +232,9 @@ Status PartitionedAggreg
EmmyMiao87 commented on a change in pull request #1737: Unify the msg of
'Memory exceed limit'
URL: https://github.com/apache/incubator-doris/pull/1737#discussion_r319976714
##
File path: be/src/runtime/plan_fragment_executor.cpp
##
@@ -514,7 +514,7 @@ void PlanFragmentExe
imay merged pull request #1737: Unify the msg of 'Memory exceed limit'
URL: https://github.com/apache/incubator-doris/pull/1737
This is an automated message from the Apache Git Service.
To respond to the message, please log o
lxqfy opened a new pull request #1739: Added MIN/MAX aggreate function
compatible with char/varchar
URL: https://github.com/apache/incubator-doris/pull/1739
This is an automated message from the Apache Git Service.
To respon
imay merged pull request #1718: Basic implementation for BetaRowsetReader
URL: https://github.com/apache/incubator-doris/pull/1718
This is an automated message from the Apache Git Service.
To respond to the message, please lo
gaodayue opened a new pull request #1740: use RowsetFactory to create and init
RowsetWriter
URL: https://github.com/apache/incubator-doris/pull/1740
The PR contains the following changes
- a factory method `create_rowset_writer` is added to `RowsetFactory`. All
clients of RowsetWrit
yiguolei opened a new pull request #1741: Not add alter task to tablet in alter
tablet request v2
URL: https://github.com/apache/incubator-doris/pull/1741
1. Not add alter task to tablet in alter tablet request v2
2. Should not check tablet under schema change when do compaction.
--
morningman merged pull request #1741: Not add alter task to tablet in alter
tablet request v2
URL: https://github.com/apache/incubator-doris/pull/1741
This is an automated message from the Apache Git Service.
To respond to t
morningman commented on a change in pull request #1734: Commit kafka offset in
routine load
URL: https://github.com/apache/incubator-doris/pull/1734#discussion_r320282904
##
File path: be/src/runtime/routine_load/routine_load_task_executor.cpp
##
@@ -227,6 +227,28 @@ void
morningman commented on a change in pull request #1734: Commit kafka offset in
routine load
URL: https://github.com/apache/incubator-doris/pull/1734#discussion_r320282365
##
File path: be/src/runtime/routine_load/routine_load_task_executor.cpp
##
@@ -227,6 +227,28 @@ void
morningman commented on a change in pull request #1739: Added MIN/MAX aggregate
function compatible with char/varchar
URL: https://github.com/apache/incubator-doris/pull/1739#discussion_r320286176
##
File path: be/src/olap/aggregate_func.h
##
@@ -203,6 +203,36 @@ struct Ag
morningman commented on a change in pull request #1739: Added MIN/MAX aggregate
function compatible with char/varchar
URL: https://github.com/apache/incubator-doris/pull/1739#discussion_r320285662
##
File path: be/src/olap/aggregate_func.h
##
@@ -203,6 +203,36 @@ struct Ag
lxqfy commented on a change in pull request #1739: Added MIN/MAX aggregate
function compatible with char/varchar
URL: https://github.com/apache/incubator-doris/pull/1739#discussion_r320331794
##
File path: be/src/olap/aggregate_func.h
##
@@ -203,6 +203,36 @@ struct Aggrega
lxqfy commented on a change in pull request #1739: Added MIN/MAX aggregate
function compatible with char/varchar
URL: https://github.com/apache/incubator-doris/pull/1739#discussion_r320332527
##
File path: be/src/olap/aggregate_func.h
##
@@ -203,6 +203,36 @@ struct Aggrega
morningman commented on a change in pull request #1739: Added MIN/MAX aggregate
function compatible with char/varchar
URL: https://github.com/apache/incubator-doris/pull/1739#discussion_r320539275
##
File path: be/src/olap/aggregate_func.h
##
@@ -203,6 +203,36 @@ struct Ag
imay opened a new pull request #1742: Reduce unnecessary memory allocat and
copy in OlapScanNode
URL: https://github.com/apache/incubator-doris/pull/1742
This is an automated message from the Apache Git Service.
To respond t
imay commented on a change in pull request #1740: use RowsetFactory to create
and init RowsetWriter
URL: https://github.com/apache/incubator-doris/pull/1740#discussion_r320557618
##
File path: be/src/olap/merger.cpp
##
@@ -21,107 +21,55 @@
#include
#include "olap/olap
imay commented on a change in pull request #1738: add bloom filter index
URL: https://github.com/apache/incubator-doris/pull/1738#discussion_r320569559
##
File path: be/src/olap/bloom_filter.hpp
##
@@ -71,6 +70,16 @@ class BitSet {
return true;
}
+bool i
imay commented on a change in pull request #1738: add bloom filter index
URL: https://github.com/apache/incubator-doris/pull/1738#discussion_r320571323
##
File path: be/src/olap/rowset/segment_v2/bloom_filter_page.cpp
##
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software
imay commented on a change in pull request #1738: add bloom filter index
URL: https://github.com/apache/incubator-doris/pull/1738#discussion_r320570534
##
File path: be/src/olap/rowset/segment_v2/bloom_filter_page.cpp
##
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software
imay commented on a change in pull request #1738: add bloom filter index
URL: https://github.com/apache/incubator-doris/pull/1738#discussion_r320569013
##
File path: be/src/olap/types.h
##
@@ -74,6 +74,7 @@ class TypeInfo {
inline void set_to_min(void* buf) const { _se
imay commented on a change in pull request #1738: add bloom filter index
URL: https://github.com/apache/incubator-doris/pull/1738#discussion_r320570400
##
File path: be/src/olap/rowset/segment_v2/bloom_filter_page.cpp
##
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software
imay commented on a change in pull request #1734: Commit kafka offset in
routine load
URL: https://github.com/apache/incubator-doris/pull/1734#discussion_r320572734
##
File path: be/src/runtime/routine_load/data_consumer.h
##
@@ -133,6 +133,8 @@ class KafkaDataConsumer : p
morningman merged pull request #1688: Fixbug： It can not create a new partition
when paritions of a table is empty
URL: https://github.com/apache/incubator-doris/pull/1688
This is an automated message from the Apache Git Ser
gaodayue commented on a change in pull request #1740: use RowsetFactory to
create and init RowsetWriter
URL: https://github.com/apache/incubator-doris/pull/1740#discussion_r320587846
##
File path: be/src/olap/merger.cpp
##
@@ -21,107 +21,55 @@
#include
#include "olap/
imay commented on issue #1726: CPU idle is still high when concurrent access is
high
URL:
https://github.com/apache/incubator-doris/issues/1726#issuecomment-527759718
After looking into this case, I found some problems that will hurt
performance.
1. There are too many unnecessary m
imay opened a new issue #1743: compute skew when concurrent queries
URL: https://github.com/apache/incubator-doris/issues/1743
When we do high concurrency test to Doris, we found that one of all backends
are complete consumed, but other backends have 60% CPU idle.
*\`
SingleQuoteString
kangpinghuang opened a new pull request #1745: add delete predicate
filter(#1636)
URL: https://github.com/apache/incubator-doris/pull/1745
Delete predicate can be used to prune data by zone map.
This is an automated message f
kangpinghuang commented on issue #1652: support filter with predicate in
BetaRowset
URL:
https://github.com/apache/incubator-doris/issues/1652#issuecomment-527873905
I will do this feature
This is an automated message from t
imay commented on a change in pull request #1745: add delete predicate
filter(#1636)
URL: https://github.com/apache/incubator-doris/pull/1745#discussion_r320726868
##
File path: be/src/olap/delete_handler.cpp
##
@@ -343,5 +343,17 @@ void DeleteHandler::finalize() {
_i
chaoyli merged pull request #1742: Reduce unnecessary memory allocate and copy
in OlapScanNode
URL: https://github.com/apache/incubator-doris/pull/1742
This is an automated message from the Apache Git Service.
To respond to
kangkaisen opened a new issue #1746: BE core when to_bitmap argument is invalid
URL: https://github.com/apache/incubator-doris/issues/1746
Today, BE core when our user test the bitmap_union:
```
terminate called after throwing an instance of 'std::out_of_range'
what(): sto
kangkaisen opened a new pull request #1747: Add check for to_bitmap function
argument
URL: https://github.com/apache/incubator-doris/pull/1747
For https://github.com/apache/incubator-doris/issues/1746
1 Add invalid_argument and out_of_range check for to_bitmap function
2 change th
morningman commented on issue #1723: 元数据支持存储计算分离
URL:
https://github.com/apache/incubator-doris/issues/1723#issuecomment-527897477
>
我们需要把所有的节点都打平么？感觉BE、FE、Broker这些之间似乎也没有什么共性。还有打平的话我觉得叫做资源不太恰当。毕竟通常我们说的都是CPU资源，IO资源，网络资源，存储资源，一般都指的是逻辑的概念。
这个我得意思主要是所有的这些节点都可以用一套 tag 机制来管理。而当前其实我们主要是对
morningman commented on issue #1723: 元数据支持存储计算分离
URL:
https://github.com/apache/incubator-doris/issues/1723#issuecomment-527897856
资源这个词，可以再讨论~
This is an automated message from the Apache Git Service.
To respond to the messa
kangkaisen commented on issue #1723: 元数据支持存储计算分离
URL:
https://github.com/apache/incubator-doris/issues/1723#issuecomment-527907152
> CREATE TABLE tbl1 (k1 int) DISTRIBUTED BY HASH(k1) BUCKETS 4
PROPERTIES (
"backend_tag" = "rack1:2, rack2:1,", // required
"remote_storage_ta
gaodayue opened a new pull request #1748: RowsetFactory as a single entry for
Rowset creation
URL: https://github.com/apache/incubator-doris/pull/1748
In this PR, all creations of Rowset are done through the
`RowsetFactory.create_rowset` factory method (except OlapSnapshotConverter
which
imay commented on a change in pull request #1747: Add check for to_bitmap
function argument
URL: https://github.com/apache/incubator-doris/pull/1747#discussion_r320787795
##
File path: be/src/exec/base_scanner.cpp
##
@@ -144,6 +144,14 @@ bool BaseScanner::fill_dest_tuple(c
imay commented on a change in pull request #1748: RowsetFactory as a single
entry for Rowset creation
URL: https://github.com/apache/incubator-doris/pull/1748#discussion_r320795386
##
File path: be/src/olap/task/engine_clone_task.cpp
##
@@ -847,10 +844,19 @@ OLAPStatus
En
imay commented on issue #1723: 元数据支持存储计算分离
URL:
https://github.com/apache/incubator-doris/issues/1723#issuecomment-527934459
> > CREATE TABLE tbl1 (k1 int) DISTRIBUTED BY HASH(k1) BUCKETS 4
> > PROPERTIES (
> > "backend_tag" = "rack1:2, rack2:1,", // required
> > "remote_storage_ta
imay closed issue #1723: 元数据支持存储计算分离
URL: https://github.com/apache/incubator-doris/issues/1723
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abov
morningman opened a new issue #1723: 元数据支持存储计算分离
URL: https://github.com/apache/incubator-doris/issues/1723
# 元数据支持存储计算分离
在存储计算架构中，数据可能存储在本地、远端，或同时存储在本地和远端。当数据存储在远端时，本地存储将作为一个全量Cache存在。在这种架构，现有的以下流程需求进行重新设计：
* Backend 的元数据管理
* 建表或创建分区时数据分片的分布管理
* 数据副本的修改，数据分片的修复和均衡
*
imay removed a comment on issue #1723: 元数据支持存储计算分离
URL:
https://github.com/apache/incubator-doris/issues/1723#issuecomment-527934459
> > CREATE TABLE tbl1 (k1 int) DISTRIBUTED BY HASH(k1) BUCKETS 4
> > PROPERTIES (
> > "backend_tag" = "rack1:2, rack2:1,", // required
> > "remote_st
morningman commented on issue #1723: 元数据支持存储计算分离
URL:
https://github.com/apache/incubator-doris/issues/1723#issuecomment-528150030
> 要求用户在建表时必须指定backend_tag，感觉不是很合理。
我觉得我们可以在整个集群，cluster，DB粒度设置backend_tag属性，建表时backend_tag作为可选项。
让用户在建表时指定backend_tag，会极大增加Doris Admin的运维成本。
backend_ta
gaodayue commented on a change in pull request #1748: RowsetFactory as a single
entry for Rowset creation
URL: https://github.com/apache/incubator-doris/pull/1748#discussion_r321049846
##
File path: be/src/olap/task/engine_clone_task.cpp
##
@@ -847,10 +844,19 @@ OLAPStatus
kangkaisen commented on a change in pull request #1747: Add check for to_bitmap
function argument
URL: https://github.com/apache/incubator-doris/pull/1747#discussion_r321050410
##
File path: be/src/exec/base_scanner.cpp
##
@@ -144,6 +144,14 @@ bool BaseScanner::fill_dest_t
imay merged pull request #1748: RowsetFactory as a single entry for Rowset
creation
URL: https://github.com/apache/incubator-doris/pull/1748
This is an automated message from the Apache Git Service.
To respond to the message
imay merged pull request #1747: Add check for to_bitmap function argument
URL: https://github.com/apache/incubator-doris/pull/1747
This is an automated message from the Apache Git Service.
To respond to the message, please lo
imay opened a new pull request #1749: Fix compile error
URL: https://github.com/apache/incubator-doris/pull/1749
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
imay merged pull request #1749: Fix compile error
URL: https://github.com/apache/incubator-doris/pull/1749
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
yiguolei opened a new pull request #1750: Check rowset is useful atomicly
URL: https://github.com/apache/incubator-doris/pull/1750
This is an automated message from the Apache Git Service.
To respond to the message, please lo
yiguolei opened a new issue #1751: Active rowset is garbage collected
URL: https://github.com/apache/incubator-doris/issues/1751
Rowset is published but is garbaged collected during gc thread.
This is an automated message from
yiguolei commented on issue #1750: Check rowset is useful atomicly
URL: https://github.com/apache/incubator-doris/pull/1750#issuecomment-528691384
#1751
This is an automated message from the Apache Git Service.
To respond to t
yiguolei commented on issue #1751: Active rowset is garbage collected
URL:
https://github.com/apache/incubator-doris/issues/1751#issuecomment-528691857
During publish version stage, it will first add rowset meta to tablet meta
and then remove it from rowset meta manager. If we check table
imay opened a new issue #1752: Create FieldList to reuse filed and schema
URL: https://github.com/apache/incubator-doris/issues/1752
Now in our storage engine, we create so many Schema/Field for many purpose.
I think after a table is created, its Schema and Filed are not modified unless
th
kangpinghuang commented on a change in pull request #1738: add bloom filter
index
URL: https://github.com/apache/incubator-doris/pull/1738#discussion_r321569645
##
File path: be/src/olap/rowset/segment_v2/bloom_filter_page.cpp
##
@@ -0,0 +1,114 @@
+// Licensed to the Apach
gaodayue opened a new pull request #1753: Use crc32c checksum for segment v2
URL: https://github.com/apache/incubator-doris/pull/1753
The following error is encountered when testing V2 table.
```
Page checksum mismatch, actual=3372108349 vs expect=3375987274
```
The reasons a
gaodayue opened a new pull request #1754: fix a bug in BetaRowsetReader which
results in empty result
URL: https://github.com/apache/incubator-doris/pull/1754
I didn't notice that `Reader` [relies
on](https://github.com/apache/incubator-doris/blob/master/be/src/olap/reader.cpp#L120)
the `
imay commented on issue #1753: Use crc32c checksum for segment v2
URL: https://github.com/apache/incubator-doris/pull/1753#issuecomment-528724219
@gaodayue
I have two questions about this PR.
1. Even if HashUtils::crc_hash doesn't generate standard CRC-32 checksum,
the checksums
imay opened a new issue #1755: Remove author tag in our code
URL: https://github.com/apache/incubator-doris/issues/1755
We should remove author tag in our code
[Here](http://www.apache.org/foundation/records/minutes/2004/board_minutes_2004_09_22.txt)
and [Here] is reference.
--
gaodayue commented on issue #1753: Use crc32c checksum for segment v2
URL: https://github.com/apache/incubator-doris/pull/1753#issuecomment-528738158
@imay comments inline
> Even if HashUtils::crc_hash doesn't generate standard CRC-32 checksum, the
checksums should be equal. Because
imay merged pull request #1754: fix a bug in BetaRowsetReader which results in
empty result
URL: https://github.com/apache/incubator-doris/pull/1754
This is an automated message from the Apache Git Service.
To respond to the
3201 - 3300 of 7706 matches
Mail list logo
