kangkaisen opened a new issue #2866: [Proposal] Support the third type Table(Partition) properties modification URL: https://github.com/apache/incubator-doris/issues/2866 ## 1 The three type Table(Partition) properties modification ### 1.1 Properties only persistent in FE metadata Such as `replication_num`, `colocate_with`, `dynamic_partition` , and so on. When modify this type properties, when only need modify these properties in FE memory metadata and insert a edit log to `BDBJEJournal`, which is very easy. ### 1.2 Properties persistent in BE tablet metadata and affect the segment file content. Such as `bloom_filter_columns`, `bitmap_inverted_index`, and so on. When modify this type properties, we need firstly create new tablet according to new properties, and then, we will need to change the segment file content by `SchemaChangeJob` ### 1.3 Properties persistent in BE tablet metadata but don't affect the segment file content. Such as `in_memory` in https://github.com/apache/incubator-doris/issues/2846 ## 2. How to Support third type properties modification ### 2.1 Handle it by `SchemaChangeJob` Like the second type properties modification, we need create new tablet according to new properties, and then, we change the segment file content by `SchemaChangeJob`, In BE, we could use `LinkedSchemaChange`. **Although the new tablet creation and segment file link both are unnecessary, but which should be work.** **Unfortunately, This way couldn't work for Partition properties modification, why?** When we create new tablet, we need to generate new SHADOW index id ``` long shadowIndexId = catalog.getNextId(); // create SHADOW index for each partition for (Partition partition : olapTable.getPartitions()) { ``` And currently, we assume MaterializedIndex ids in all partitions are all the same. ``` public long selectBestRollup( Collection<Long> partitionIds, List<Expr> conjuncts, boolean isPreAggregation) throws UserException { Preconditions.checkArgument(partitionIds != null && !partitionIds.isEmpty(), "Paritition can't be null or empty."); // Get first partition to select best prefix index rollups, because MaterializedIndex ids in one rollup's partitions are all same. ``` **So, If we only update one partition MaterializedIndex id, when query, there will be a error.** So, we shouldn't use this solution. ### 2.1 Only update the tablet meta by a agent task Directly update the BE tablet meta, and don't need to create new tablet and link the segment file. ## 3 Detailed design ### 3.1 FE 1 Add `TTabletMetaType` Field to `TTabletMetaInfo`, we could update different tablet meta according to `TTabletMetaType`. 2 Make the `UpdateTabletMetaInfoTask` support synchronous process by `MarkedCountDownLatch` like `CreateReplicaTask` 3 Handle `UpdateTabletMetaInfoTask` result in `MasterImpl` 4 After `UpdateTabletMetaInfoTask` success, update the FE metadata by editlog. ### 3.2 BE 1 Add `set_is_in_memory` method to `TabletSchema` 2 Remove const qualifier for `TabletSchema` in `tablet_meta.h` 3 Handle different tablet meta update task by different `TTabletMetaType`
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
