+1 (non-binding) On Mon, Apr 6, 2026 at 11:10 AM John Zhuge <[email protected]> wrote:
> +1 (non-binding) > > On Mon, Apr 6, 2026 at 11:07 AM Holden Karau <[email protected]> > wrote: > >> +1 (binding) >> >> On Mon, Apr 6, 2026 at 10:58 AM Anish Shrigondekar via dev < >> [email protected]> wrote: >> >>> +1 >>> >>> Thanks, >>> Anish >>> >>> On Mon, Apr 6, 2026 at 10:57 AM DB Tsai <[email protected]> wrote: >>> >>>> +1 (binding) >>>> >>>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >>>> >>>> On Apr 3, 2026, at 2:59 PM, Andreas Neumann <[email protected]> wrote: >>>> >>>> Hi Spark devs, >>>> >>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache >>>> Spark* >>>> Motivation >>>> >>>> With the upcoming introduction of standardized CDC support >>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will soon >>>> have a unified way to produce change data feeds. However, consuming these >>>> feeds and applying them to a target table remains a significant challenge. >>>> >>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and SCD >>>> Type 2 (tracking full change history) often require hand-crafted, >>>> complex MERGE logic. In distributed systems, these implementations are >>>> frequently error-prone when handling deletions or out-of-order data. >>>> Proposal >>>> >>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It >>>> encapsulates the complex logic for SCD types and out-of-order data, >>>> allowing data engineers to configure a declarative flow instead of writing >>>> manual MERGE statements. This feature will be available in both Python >>>> and SQL. >>>> Example SQL: >>>> -- Produce a change feed >>>> CREATE STREAMING TABLE cdc.users AS >>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10; >>>> >>>> -- Consume the change feed >>>> CREATE FLOW flow >>>> AS AUTO CDC INTO >>>> target >>>> FROM stream(cdc_data.users) >>>> KEYS (userId) >>>> APPLY AS DELETE WHEN operation = "DELETE" >>>> SEQUENCE BY sequenceNum >>>> COLUMNS * EXCEPT (operation, sequenceNum) >>>> STORED AS SCD TYPE 2 >>>> TRACK HISTORY ON * EXCEPT (city); >>>> >>>> *Relevant Links:* >>>> >>>> - SPIP Document: >>>> >>>> https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/ >>>> - *Discussion Thread: * >>>> https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7 >>>> - >>>> >>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668> >>>> https://issues.apache.org/jira/browse/SPARK-56249 >>>> >>>> *The vote will be open for at least 72 hours. *Please vote: >>>> >>>> [ ] +1: Accept the proposal as an official SPIP >>>> [ ] +0 >>>> [ ] -1: I don't think this is a good idea because ... >>>> Cheers -Andreas >>>> >>>> >>>> >>>> >> >> -- >> Twitter: https://twitter.com/holdenkarau >> Fight Health Insurance: https://www.fighthealthinsurance.com/ >> <https://www.fighthealthinsurance.com/?q=hk_email> >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> Pronouns: she/her >> > > > -- > John Zhuge > -- Liu Cao
