suxiaogang223 opened a new issue, #67337: URL: https://github.com/apache/doris/issues/67337
## Background Apache Paimon exposes operational table maintenance through procedures such as compaction, snapshot expiration, rollback, orphan-file cleanup, and Branch/Tag actions. Doris currently has no Catalog-routed procedure framework for invoking these Paimon operations. This issue tracks procedure execution and operational closure around the write capabilities in #65086. Branch/Tag domain behavior and branch writes are tracked separately; Branch/Tag procedure aliases must reuse that implementation. Upstream references: - https://paimon.apache.org/docs/1.4/spark/procedures/ - https://paimon.apache.org/docs/1.4/maintenance/manage-snapshots/ - https://paimon.apache.org/docs/1.4/maintenance/dedicated-compaction/ ## Current Status - Doris `CALL` supports built-in functions and Doris stored procedures, but does not resolve procedures through an External Catalog. - Paimon/Spark-style named arguments such as `table => 'db.tbl'` are not supported by the Doris `CALL` grammar. - Paimon maintenance operations must currently be run from another engine or SDK. - Doris write transactions can create snapshots and files, but operators cannot complete the corresponding maintenance lifecycle from Doris SQL. ## Goals ### Catalog Procedure Framework - Support Catalog-qualified calls such as `CALL <catalog>.sys.<procedure>(...)`. - Support positional and named arguments with procedure-specific type validation and defaults. - Resolve the target Catalog, database, table, branch, tag, and snapshot without silently falling back to another object. - Route execution through the configured Paimon Catalog and authentication context. - Return structured result sets and stable error codes/messages. - Define FE forwarding, timeout, cancellation, retry, and synchronous/asynchronous execution behavior. - Add privilege checks, audit records, sensitive-property masking, and operation metrics. The framework should be reusable by other External Catalog implementations, while this issue's functional scope is Paimon. ### Initial Paimon Procedures P0 operational closure: - `compact` - `expire_snapshots` - `remove_orphan_files` - `rollback_to` - `rollback_to_as_latest` P1 compatibility and lifecycle aliases: - `create_tag`, `delete_tag`, and `rename_tag` - `create_branch`, `delete_branch`, and `rename_branch` - `fast_forward` and `merge_branch` These Branch/Tag aliases must call the implementation owned by the Branch/Tag lifecycle issue rather than introducing a second metadata path. P2 extended operations: - Database-level compaction. - Consumer maintenance. - Partition maintenance such as mark-done or expiration actions supported by the selected Paimon version. ## Safety and Consistency Requirements - Pin the table/ref/snapshot generation used for validation and execution. - Detect concurrent commits and incompatible branch ancestry before destructive operations. - Clearly distinguish `rollback_to`, which can remove later snapshots/tags, from `rollback_to_as_latest`, which publishes a new latest state. - Expose affected snapshots/files and whether an operation made changes. - Prevent orphan cleanup from deleting files referenced by active snapshots, tags, branches, or in-flight Doris transactions. - Refresh Doris metadata caches after every metadata-changing procedure. - Define bounded resource use for large compaction and cleanup operations. ## Acceptance Criteria - Catalog-qualified procedures work against filesystem, HMS, and REST Paimon catalogs. - Positional and named argument forms are validated consistently. - Authorization and audit behavior are covered by regression tests. - Concurrent write, retry, timeout, cancellation, and partial-failure cases do not produce ambiguous silent success. - Procedure results are deterministic enough for automation and include the affected object/state. - Cross-engine validation confirms that Doris maintenance results remain readable by Spark and Flink. ## Non-goals - Paimon Branch DML implementation; it is tracked separately. - General-purpose user-defined stored procedures. - Long-running job scheduling beyond the execution model required by the initial Paimon operations. ## Related - Paimon write tracking: #65086 - Paimon Branch/Tag lifecycle and branch writes: #67336 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
