Lchangliang opened a new issue, #10135: URL: https://github.com/apache/incubator-doris/issues/10135
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description Background Add/drop column are heightly operators. They will create new tablet and rewrite the data from old tablet. When add/drop column frequently, a lot of time is wasted waiting. So we need a new way to optimize the process. Improvement This improvement involves three aspects, read, writer, compaction. In original impl, BE will hold the tablet schema, set unique id for each column. When read/writer/compaction, BE can get the schema from tablet meta. The core of the modification is that get the schema from FE when read/writer. And Every rowset will hold its schema. Using the newest schema (largest verison rowset) when doing compaction. Modification 1. Generate Unique ID by FE. 2. When reading/inserting data, FE will send the newest schema to BE. 3. When inserting, BE will persistent the schema with rowset meta. 4. When doing compaction, BE will choose newest schema from compation rowsets and make it persistent with new rowset meta after compaction. 5. The improvement is only acting on add/drop value. If add/drop key, it will be done by the old way. 6. It will compatible with old table. Old table is mean that system already has tables before the upgrade. But old table will always do the change by old way although the column is value. Result When add/drop value column, they will be lightweight operators. They don't need rewrite the data and complete quickly. ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
