List partition
Doris currently only supports Range partitioning, where data is usually
partitioned by time columns.
However, in some scenarios, users want to partition by some enumerated
values of columns, such as by city, etc.
Design
To add support for List partitioning, the following functional points need
to be considered.
1. Support for List partition syntax in creating table statements.
2. Support for adding and deleting List partition syntax.
3. Support for List partitioning in various load operations.
4. Support for List partition pruning during query.
List partitioned tables do not need to consider dynamic partitioning.
Detailed designSyntax
The main changes involved here include.
1. Implementation of the subclass ListPartitionDesc of the parsing class
PartitionDesc
2. Implementation of metadata class PartitionInfo subclass
ListPartitionInfo
3. Support for parsing and checking ListPartitionDesc in CreateTableStmt
4. Support for the creation of List Partition tables in Catalog class.
5. Metadata persistence-related changes.
The syntax is referenced from MySQL and Oracle
Single partition column
CREATE TABLE tb1 (
k1 int, k2 varchar(128), k3 int, v1 int, v2 int
)
PARTITION BY LIST(k1)
(
PARTITION p1 VALUES IN ("1", "3", "5"),
PARTITION p2 VALUES IN ("2", "4", "6"),
...
)
...
;
Multi-partition columns
CREATE TABLE tb2 (
k1 int, k2 varchar(128), k3 int, v1 int, v2 int
)
PARTITION BY LIST(k1, k2)
(
PARTITION p1 VALUES IN (("1", "beijing"), ("1", "shanghai")),
PARTITION p2 VALUES IN (("2", "beijing"), ("2", "shanghai"), ("2",
"tianjin")),
PARTITION p3 VALUES IN (("3", "beijing")),
...
)
...
;
NOTE: Each partition needs to ensure that the partition values are unique.
Add partition
ALTER TABLE tb1 ADD PARTITION p4 VALUES IN ("7", "8", "9");
ALTER TABLE tb2 ADD PARTITION p4 VALUES IN (("4", "tianjin"));
Load
The current load methods of Doris include Stream Load, INSERT, Routine
Load, Broker Load, Hadoop Load, Spark Load.
Among them, Stream Load, INSERT, Routine Load, and Broker Load all use
TabletSink class for data distribution. Our first phase supports List
partition support for these load operations.
The main changes involved include:
1. Changes related to the Descriptors.TOlapTablePartitionParam structure
in the Thrift structure TOlapTableSink
2. Changes related to the OlapTablePartition object in the OlapTableSink
class on the BE side.
Query
The query mainly needs to implement the List Partition pruning function.
The main changes involved include:
1. Implementing the subclass ListPartitionPruner of PartitionPruner
Partition related
Support operations related to partitioned tables, such as recover,
truncate, temporary partition, restore, replace, etc.
