Thank you all. I will close the vote now. On Tue, Apr 7, 2026 at 8:50 AM Dongjoon Hyun <[email protected]> wrote:
> +1 > > Dongjoon > > On 2026/04/07 07:18:19 Kousuke Saruta wrote: > > +1 > > > > > > 2026年4月7日(火) 16:17 Peter Toth <[email protected]>: > > > > > +1 (non-binding) > > > > > > On Mon, Apr 6, 2026 at 10:13 PM Jungtaek Lim < > [email protected]> > > > wrote: > > > > > >> +1 (non-binding) > > >> > > >> On Tue, Apr 7, 2026 at 3:11 AM Liu Cao <[email protected]> wrote: > > >> > > >>> +1 (non-binding) > > >>> > > >>> On Mon, Apr 6, 2026 at 11:10 AM John Zhuge <[email protected]> > wrote: > > >>> > > >>>> +1 (non-binding) > > >>>> > > >>>> On Mon, Apr 6, 2026 at 11:07 AM Holden Karau < > [email protected]> > > >>>> wrote: > > >>>> > > >>>>> +1 (binding) > > >>>>> > > >>>>> On Mon, Apr 6, 2026 at 10:58 AM Anish Shrigondekar via dev < > > >>>>> [email protected]> wrote: > > >>>>> > > >>>>>> +1 > > >>>>>> > > >>>>>> Thanks, > > >>>>>> Anish > > >>>>>> > > >>>>>> On Mon, Apr 6, 2026 at 10:57 AM DB Tsai <[email protected]> > wrote: > > >>>>>> > > >>>>>>> +1 (binding) > > >>>>>>> > > >>>>>>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 > > >>>>>>> > > >>>>>>> On Apr 3, 2026, at 2:59 PM, Andreas Neumann <[email protected]> > wrote: > > >>>>>>> > > >>>>>>> Hi Spark devs, > > >>>>>>> > > >>>>>>> I'd like to call a vote on the SPIP*: Auto CDC Support for Apache > > >>>>>>> Spark* > > >>>>>>> Motivation > > >>>>>>> > > >>>>>>> With the upcoming introduction of standardized CDC support > > >>>>>>> <https://issues.apache.org/jira/browse/SPARK-55668>, Spark will > > >>>>>>> soon have a unified way to produce change data feeds. However, > > >>>>>>> consuming these feeds and applying them to a target table > remains a > > >>>>>>> significant challenge. > > >>>>>>> > > >>>>>>> Common patterns like SCD Type 1 (maintaining a 1:1 replica) and > SCD > > >>>>>>> Type 2 (tracking full change history) often require hand-crafted, > > >>>>>>> complex MERGE logic. In distributed systems, these > implementations > > >>>>>>> are frequently error-prone when handling deletions or > out-of-order data. > > >>>>>>> Proposal > > >>>>>>> > > >>>>>>> This SPIP proposes a new "Auto CDC" flow type for Spark. It > > >>>>>>> encapsulates the complex logic for SCD types and out-of-order > data, > > >>>>>>> allowing data engineers to configure a declarative flow instead > of writing > > >>>>>>> manual MERGE statements. This feature will be available in both > Python > > >>>>>>> and SQL. > > >>>>>>> Example SQL: > > >>>>>>> -- Produce a change feed > > >>>>>>> CREATE STREAMING TABLE cdc.users AS > > >>>>>>> SELECT * FROM STREAM my_table CHANGES FROM VERSION 10; > > >>>>>>> > > >>>>>>> -- Consume the change feed > > >>>>>>> CREATE FLOW flow > > >>>>>>> AS AUTO CDC INTO > > >>>>>>> target > > >>>>>>> FROM stream(cdc_data.users) > > >>>>>>> KEYS (userId) > > >>>>>>> APPLY AS DELETE WHEN operation = "DELETE" > > >>>>>>> SEQUENCE BY sequenceNum > > >>>>>>> COLUMNS * EXCEPT (operation, sequenceNum) > > >>>>>>> STORED AS SCD TYPE 2 > > >>>>>>> TRACK HISTORY ON * EXCEPT (city); > > >>>>>>> > > >>>>>>> *Relevant Links:* > > >>>>>>> > > >>>>>>> - SPIP Document: > > >>>>>>> > https://docs.google.com/document/d/1Hp5BGEYJRHbk6J7XUph3bAPZKRQXKOuV1PEaqZMMRoQ/ > > >>>>>>> - *Discussion Thread: * > > >>>>>>> > https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7 > > >>>>>>> - > > >>>>>>> > > >>>>>>> JIRA: <https://issues.apache.org/jira/browse/SPARK-55668> > > >>>>>>> https://issues.apache.org/jira/browse/SPARK-56249 > > >>>>>>> > > >>>>>>> *The vote will be open for at least 72 hours. *Please vote: > > >>>>>>> > > >>>>>>> [ ] +1: Accept the proposal as an official SPIP > > >>>>>>> [ ] +0 > > >>>>>>> [ ] -1: I don't think this is a good idea because ... > > >>>>>>> Cheers -Andreas > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>> > > >>>>> -- > > >>>>> Twitter: https://twitter.com/holdenkarau > > >>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ > > >>>>> <https://www.fighthealthinsurance.com/?q=hk_email> > > >>>>> Books (Learning Spark, High Performance Spark, etc.): > > >>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > > >>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau > > >>>>> Pronouns: she/her > > >>>>> > > >>>> > > >>>> > > >>>> -- > > >>>> John Zhuge > > >>>> > > >>> > > >>> > > >>> -- > > >>> > > >>> Liu Cao > > >>> > > >>> > > >>> > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > >
