[
https://issues.apache.org/jira/browse/FLINK-39062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
featzhang updated FLINK-39062:
------------------------------
Summary: Support applying watermark on logical relations in SQL (was:
Support logical-level watermark definition via SQL function)
> Support applying watermark on logical relations in SQL
> ------------------------------------------------------
>
> Key: FLINK-39062
> URL: https://issues.apache.org/jira/browse/FLINK-39062
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API, Table SQL / Planner
> Reporter: featzhang
> Priority: Major
>
> h1. 1. Motivation
> Currently, watermark definition in Flink SQL is tightly coupled with table
> DDL. This introduces several limitations:
> # Watermark is treated as catalog metadata instead of a logical relational
> property.
> # It cannot be flexibly reused across logical abstractions (e.g., view-based
> pipelines).
> # It complicates modular query design when watermark logic needs to be
> shared.
> During earlier discussion, extending VIEW DDL to support watermark was
> considered. However, this approach raised concerns around:
> * Semantic ambiguity between metadata and logical properties
> * Increased complexity in view expansion and optimization
> * Potential impact on existing DDL semantics
> To address these concerns while still enabling logical reuse of watermark
> definitions, this FLIP proposes introducing a function-based approach.
> ----
> h1. 2. Proposed Solution
> Introduce a built-in table function:
>
> {code:java}
> APPLY_WATERMARK(
> table,
> DESCRIPTOR(rowtime_column),
> watermark_expression
> ){code}
>
> This function attaches watermark semantics at the logical relational level
> instead of the catalog level.
> ----
> h1. 3. Syntax
>
>
>
>
> {code:java}
> SELECT *
> FROM APPLY_WATERMARK(
> Orders,
> DESCRIPTOR(order_time),
> order_time - INTERVAL '5' SECOND
> );{code}
>
> The first argument supports:
> * Base tables
> * Views (including non-materialized views)
> * Subqueries
> This enables logical reuse without introducing watermark semantics into
> catalog DDL.
> ----
> h1. 4. Semantics
> * The function returns a relation identical to the input table but with an
> attached watermark definition.
> * The watermark is scoped to the query.
> * It does not modify catalog metadata.
> * It does not alter view definitions.
> Watermark propagation rules remain consistent with current planner behavior.
> ----
> h1. 5. Why Function-Based Instead of VIEW Extension
> The function-based approach:
> * Keeps watermark as a logical property rather than catalog metadata
> * Avoids modifying VIEW DDL semantics
> * Avoids ambiguity between physical and logical properties
> * Aligns with existing function-style relational extensions
> * Minimizes impact on optimizer and catalog layers
> This design keeps the scope focused and reduces planner and metadata coupling.
> ----
> h1. 6. Design Evolution
> The initial proposal considered allowing watermark definition directly inside
> VIEW DDL to support reuse.
> After discussion, it was identified that:
> * Watermark semantics are fundamentally logical rather than metadata.
> * Embedding watermark in VIEW DDL introduces complexity in view expansion.
> * It may blur the boundary between logical and catalog layers.
> Based on community feedback, the proposal pivots to a function-based solution
> that preserves logical reuse while avoiding DDL-level changes.
> The view-based design is no longer preferred.
> ----
> h1. 7. Compatibility
> * No changes to existing table DDL behavior.
> * No backward compatibility impact.
> * Purely additive feature.
> ----
> h1. 8. Scope
> This FLIP only introduces:
> * The APPLY_WATERMARK function
> * Planner support for logical watermark attachment
> It does not:
> * Modify catalog metadata
> * Change existing VIEW DDL syntax
> * Introduce physical materialization behavior
> ----
> h1. 9. Implementation Plan
> # Add APPLY_WATERMARK function definition
> # Extend planner to attach logical watermark trait
> # Add validation rules for descriptor and expression
> # Add unit and integration tests
> ----
> h1. 10. Open Questions
> * Should nested APPLY_WATERMARK calls be allowed?
> * Should conflicting watermark definitions throw validation errors?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
