morningman opened a new pull request #3120: [Temp Partition] Support loading data into temp partitions URL: https://github.com/apache/incubator-doris/pull/3120 Related issue: #2663, #2828. This CL support loading data into specified temporary partitions. ``` INSERT INTO tbl TEMPORARY PARTITIONS(tp1, tp2, ..) ....; curl .... -H "temporary_partition: tp1, tp, .. " .... LOAD LABEL db1.label1 ( DATA INFILE("xxxx") INTO TABLE `tbl2` TEMPORARY PARTITION(tp1, tp2, ...) ... ``` ## Meta refactor In order to be able to support specifying temporary partitions, I made some changes to the way the partition information in the table is stored. Partition information is now organized as follows: The following two maps are reserved in OlapTable for storing formal partitions: ``` idToPartition nameToPartition ``` Use the `TempPartitions` class for storing temporary partitions. All the partition attributes of the formal partition and the temporary partition, such as the range, the number of replicas, and the storage medium, are all stored in the `partitionInfo` of the OlapTable. In `partitionInfo`, we use two maps to store the range of formal partition and temporary partition: ``` idToRange idToTempRange ``` Use separate map is because the partition ranges of the formal partition and the temporary partition may overlap. Separate map can more easily check the partition range. All partition attributes except the partition range are stored using the same map, and the partition id is used as the map key. ## Method to get partition A table may contain both formal and temporary partitions. There are several methods to get the partition of a table. Typically divided into two categories: 1. Get partition by id 2. Get partition by name According to different requirements, the caller may want to obtain a formal partition or a temporary partition. These methods are described below in order to obtain the partition by using the correct method. 1. Get by name This type of request usually comes from a user with partition names. Such as `select * from tbl partition(p1);`. This type of request has clear information to indicate whether to obtain a formal or temporary partition. Therefore, we need to get the partition through this method: `getPartition(String partitionName, boolean isTemp)` To avoid modifying too much code, we leave the `getPartition(String partitionName)`, which is same as: `getPartition(partitionName, false)` 2. Get by id This type of request usually means that the previous step has obtained certain partition ids in some way, so we only need to get the corresponding partition through this method: `getPartition(long partitionId)`. This method will try to get both formal partitions and temporary partitions. 3. Get all partition instances Depending on the requirements, the caller may want to obtain all formal partitions, all temporary partitions, or all partitions. Therefore we provide 3 methods, the caller chooses according to needs. `getPartitions()` `getTempPartitions()` `getAllPartitions()`
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
