suxiaogang223 opened a new issue, #67336: URL: https://github.com/apache/doris/issues/67336
## Background Apache Doris can read Apache Paimon branches and tags, including branch-specific schemas and retained tag snapshots. However, Doris cannot currently create or drop Paimon branches/tags, and Paimon branches cannot be used as Doris write targets. This issue tracks the Branch/Tag lifecycle and branch-write follow-up to #65086. Upstream reference: https://paimon.apache.org/docs/1.4/maintenance/manage-branches/ ## Current Status - Read from `table@branch(name)`: supported. - Read from `table@tag(name)`: supported. - Query `$branches` and `$tags` system tables: supported. - Create/drop Branch or Tag from Doris SQL: not supported. - Write to a Branch from Doris SQL: not supported. - Tag writes: must remain unsupported because tags are immutable references. - Fast Forward / Merge / Replace operations: not supported from Doris. ## Goals ### Branch and Tag Lifecycle - Create a branch from the current table state, a snapshot, or a tag. - Drop and rename branches. - Create a tag from the latest or a specified snapshot. - Support tag retention when the Paimon version and catalog provide it. - Drop and rename tags. - Support `IF EXISTS`, `IF NOT EXISTS`, and idempotent retry semantics where applicable. - Invalidate Doris metadata, schema, snapshot, and partition caches after reference changes. The canonical Doris interface should reuse the existing external-table Branch/Tag DDL model. Procedure aliases may be added by the separate maintenance-procedure issue, but must call the same underlying implementation. ### Branch Writes - Support `INSERT INTO` and `INSERT OVERWRITE` on a Paimon branch. - Support row-level `UPDATE`, `DELETE`, and `MERGE` on a Paimon branch where the target table supports them. - Bind analysis, distribution, writer initialization, commit, abort, and reconciliation to the selected branch table and its schema generation. - Keep transaction identity, metadata refresh, and cache keys scoped to the branch. - Reject every write to a tag during analysis. - Never fall back to `main` when branch resolution or branch commit fails. ### Promotion and Conflict Semantics - Support Fast Forward from a branch to `main`. - Define Merge / Replace behavior according to the Paimon version used by Doris. - Detect incompatible branch ancestry, schema history, and concurrent changes before promotion. - Define behavior when `main` or the branch advances during a Doris statement. - Preserve Paimon commit/abort and retry semantics on branch-targeted transactions. ### Interoperability - Read and write branches created by Flink or Spark. - Make Doris-created branches and tags visible and usable from Flink and Spark. - Cover filesystem, HMS, and REST catalogs. ## Acceptance Criteria - Branch `INSERT`, overwrite, and row-level DML mutate only the selected branch. - Main, other branches, and tags remain unchanged after a successful or failed branch write. - Tag writes fail during analysis with a clear error. - Branch schema evolution is resolved from the branch rather than the latest `main` schema. - Fast Forward succeeds only for a valid ancestry and leaves Doris caches consistent with Paimon metadata. - Commit timeout, retry, abort, FE-side reconciliation, and concurrent promotion cases have regression coverage. - Cross-engine tests cover branches/tags created and consumed by Doris, Spark, and Flink. ## Suggested Delivery Phases 1. Branch/Tag create, drop, read interoperability, and immutable Tag enforcement. 2. Branch `INSERT INTO` / `INSERT OVERWRITE` with transaction isolation. 3. Branch `UPDATE` / `DELETE` / `MERGE` and Fast Forward. 4. Merge / Replace, retention, rename, and extended concurrency coverage. ## Non-goals - A generic Catalog procedure framework. - Snapshot expiration, orphan-file cleanup, compaction scheduling, or rollback procedures. - Rust FFI write backend work. ## Related - Paimon write tracking: #65086 - Paimon maintenance procedures and actions: to be linked after creation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
