Hi > Until now ChunJun is the only tool that can capture all tables data change of > the whole database
With the above feature description, I am curious too about is this better to be an extension of Camel to provide data changes into a pipeline? Could you share why did you think `Establishing a new project and rebuilding all things from new` is better? Have you considered integrating with Camel? Sheng Wu 吴晟 Twitter, wusheng1108 LuNing Wang <wang4lun...@gmail.com> 于2022年2月25日周五 16:38写道： > > Hi, > > Thank a lot Lidong Dai 代立冬老师. > > > I am curious that Chunjun(was Flinkx) is built on the Flink CDC, so what is > > its innovation? > > The ChunJun is not built on the Flink CDC. When we built the ChunJun > CDC code, the Flink CDC did not exist. We independently created a new > data structure that is different from Flink CDC. Every CDC record of > this data structure owns all metadata. > > Until now ChunJun is the only tool that can capture all tables data > change of the whole database by configuring a simple file rather than > writing java code. The Flink CDC can capture one table by using SQL, > and capture all tables by using java code. > > We will also implement capture schema evolution, it is also based on a > new data structure. Now we have implemented capture schema evolution > in the reader/source part, we will implement the writer/sink part. > > Best, > LuNing Wang 王鲁宁 > > Jean-Baptiste Onofré <j...@nanthrax.net> 于2022年2月25日周五 14:44写道： > > > > Hi > > > > Thanks for the details. > > > > Another question: how do you compare Chunjun with camel, nifi, goblin, and > > others ? > > What kind of community you already have ? > > > > Thanks > > Regards > > JB > > > > Le ven. 25 févr. 2022 à 07:28, LuNing Wang <wang4lun...@gmail.com> a écrit : > >> > >> Hi, > >> > >> Thanks a lot Jean-Baptiste Onofré. > >> > >> > how the users implement the ETL pipeline ? > >> > >> Users implement the ETL pipeline by configuring a json file or SQL. > >> the following is one of examples: > >> https://github.com/DTStack/chunjun/blob/master/flinkx-examples/json/doris/mysql_doris.json > >> > >> > >> > Is it using a kind of SDK/DSL or do you plan an UI ? > >> > >> It is using a json file or SQL file. We had a plan to implement an UI > >> a year ago, but current core developers have no time to implement an > >> UI and are not good at Web frontend skills. > >> > >> > >> > Is it very coupled to Flink or the runtime is "pluggable" ? Do you plan > >> > to have some kind of optional storage, similar to Apache Druid. > >> > >> Nowadays, the ChunJun is coupled to Flink. We plan to support Spark or > >> other runtimes, but no person in the ChunJun community is good at > >> Apache Spark. We need to bring in more contributors. > >> > >> Best, > >> LuNing Wang 王鲁宁 > >> > >> Daniel Widdis <wid...@gmail.com> 于2022年2月25日周五 11:11写道： > >> > > >> > Hi, LuNing. > >> > > >> > I am not an IPC member, just an interested open source enthusiast > >> > looking for a project to contribute to. > >> > > >> > I was excited about contributing to another project that recently joined > >> > the incubator but as someone who only speaks English, I have had a > >> > challenge when a large number of issues are written in Chinese without > >> > enough translation for me to help. > >> > > >> > Reading your proposal I thought this may be another opportunity for me > >> > to help, but I just visited your github site and found that most of the > >> > current open issues are not understandable to me, and while I am > >> > enthusiastic and want to help, I do not see how I can do so. > >> > > >> > I am concerned about your ability to gain members of your project who > >> > do not speak your language. > >> > > >> > On 2/23/22, 9:15 PM, "LuNing Wang" <wang4lun...@gmail.com> wrote: > >> > > >> > Hi, > >> > > >> > I am LuNing Wang who sent the Chunjun proposal using > >> > 'apa...@dtstack.com' > >> > and I'm one of the maintainer of Chunjun project. > >> > As Apache is a community of peers, I will use this email to reply to > >> > all > >> > questions and issues in this thread, after I read The Apache > >> > Incubator > >> > Cookbook. > >> > > >> > May I use this email to continue to communicate with the Apache > >> > community > >> > in this thread? > >> > > >> > Best, > >> > LuNing Wang 王鲁宁 > >> > > >> > > >> > Apache <apa...@dtstack.com> 于2022年2月24日周四 09:57写道： > >> > > >> > > > >> > > > >> > > ------------------------------------------------------------------ > >> > > 发件人：Calvin Kirs <k...@apache.org> > >> > > 发送时间：2022年2月23日(星期三) 00:07 > >> > > 收件人：general <general@incubator.apache.org> > >> > > 主 题：Re: Chunjun Proposal > >> > > > >> > > Hi, > >> > > > >> > > I second with Tison and I'm glad to see your proposal, > >> > > you must have put a lot of effort into drafting this proposal, > >> > > but it needs to be clear what you expect and what you know about > >> > Apache[1], > >> > > and are we are on the same page? > >> > > > >> > > > >> > > You can see the following information: > >> > > The Apache Incubator Cookbook[2] > >> > > The Apache Way[3] > >> > > > >> > > I'd be happy to help you if you need it. > >> > > > >> > > [1]https://www.apache.org/ > >> > > [2]https://incubator.apache.org/cookbook/ > >> > > [3]https://www.apache.org/theapacheway/index.html > >> > > > >> > > tison <wander4...@gmail.com> 于2022年2月22日周二 23:17写道： > >> > > > >> > > > Hi, > >> > > > > >> > > > I have two questions here: > >> > > > > >> > > > 1. How should I name you in this thread? Apache is a community > >> > of peers. > >> > > I > >> > > > can't image I'm talking to the whole Chunjun community or > >> > "Apache" expect > >> > > > its your name. > >> > > > 2. What's your expectations on going into the incubator? Among > >> > the whole > >> > > > proposal it's almost about what the current state of Chunjun and > >> > the only > >> > > > statement about your expectations is: > >> > > > > >> > > > > we seek to further prosper the community with the aid of Apache > >> > > > > >> > > > Could you elaborate a bit the motivation here? What help are you > >> > seeking? > >> > > > > >> > > > Also I second to Sheng's comment that it's confused about your > >> > > expressions > >> > > > of contributors and initial committers. If your community > >> > continuously > >> > > > promote contributors , why the initial committer list is quite a > >> > bit > >> > > > limited? > >> > > > > >> > > > Best, > >> > > > tison. > >> > > > > >> > > > > >> > > > Sheng Wu <wu.sheng.841...@gmail.com> 于2022年2月22日周二 20:54写道： > >> > > > > >> > > > > I think this description is incorrect. > >> > > > > > >> > > > > > Our initial committers will submit iCLA(s), SGA, and CCLA(s). > >> > > > > > >> > > > > Committers are individuals, who should only submit ICLA, their > >> > > > > employers are recommended to submit CCLA, the owner of the > >> > project > >> > > > > should sign the SGA. > >> > > > > ___ > >> > > > > > >> > > > > Also, I noticed a conflict in your description > >> > > > > On one side, you mentioned `The initial committers are > >> > employees of > >> > > > > DTStack.` with only 5 initial committers, and on the other > >> > hand, you > >> > > > > gave a very long vendor list and core contributors list. > >> > > > > So, which is an accurate description? If you have those > >> > contributors, > >> > > > > why were all of them invited as PPMC members? Do you have any > >> > public > >> > > > > discussion about this decision? > >> > > > > Such as GitHub ID(demotto) is the #8 in the contributor list, > >> > and also > >> > > > > listed in the core contributor list, but can't find it in the > >> > initial > >> > > > > committer list. > >> > > > > > >> > > > > > >> > > > > Sheng Wu 吴晟 > >> > > > > Twitter, wusheng1108 > >> > > > > > >> > > > > Lidong Dai <lidong...@apache.org> 于2022年2月22日周二 20:41写道： > >> > > > > > > >> > > > > > I am curious that Chunjun(was Flinkx) is built on the Flink > >> > CDC, so > >> > > > what > >> > > > > is > >> > > > > > its innovation? > >> > > > > > > >> > > > > > BTW, you shouldn't use the mail(apa...@dtstack.com) as your > >> > user > >> > > > > account, > >> > > > > > apache is a registered trademark of ASF > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > Best Regards > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > --------------- > >> > > > > > Apache DolphinScheduler PMC Chair > >> > > > > > Lidong Dai > >> > > > > > lidong...@apache.org > >> > > > > > Linkedin: https://www.linkedin.com/in/dailidong > >> > > > > > Twitter: @WorkflowEasy <https://twitter.com/WorkflowEasy> > >> > > > > > > >> > > > > > --------------- > >> > > > > > > >> > > > > > > >> > > > > > On Tue, Feb 22, 2022 at 3:10 PM Jean-Baptiste Onofré < > >> > > j...@nanthrax.net> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > By the way, if you need a mentor, you can ping me. > >> > > > > > > > >> > > > > > > Regards > >> > > > > > > JB > >> > > > > > > > >> > > > > > > On Tue, Feb 22, 2022 at 8:08 AM Jean-Baptiste Onofré < > >> > > > j...@nanthrax.net> > >> > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > Hi, > >> > > > > > > > > >> > > > > > > > It looks interesting. Quick questions about the current > >> > state of > >> > > > the > >> > > > > > > > project/roadmap: how the users implement the ETL > >> > pipeline ? Is it > >> > > > > > > > using a kind of SDK/DSL or do you plan an UI ? > >> > > > > > > > Is it very coupled to Flink or the runtime is > >> > "pluggable" ? Do > >> > > you > >> > > > > > > > plan to have some kind of optional storage, similar to > >> > Apache > >> > > > Druid, > >> > > > > > > > ... ? > >> > > > > > > > > >> > > > > > > > Regards > >> > > > > > > > JB > >> > > > > > > > > >> > > > > > > > On Tue, Feb 22, 2022 at 7:41 AM Apache > >> > <apa...@dtstack.com> > >> > > wrote: > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > Dear Apache Incubator Community, > >> > > > > > > > > We propose to contribute Chunjun as an Apache Incubator > >> > > project. > >> > > > > > > > > We are still looking for possible Champion and Mentors > >> > if > >> > > anyone > >> > > > > would > >> > > > > > > like to volunteer. Thanks a lot. > >> > > > > > > > > Best Regards, > >> > > > > > > > > Real-time computing engine team of DTStack. > >> > > > > > > > > > >> > > > > > > > > #Chunjun Proposal > >> > > > > > > > > > >> > > > > > > > > ##Abstract > >> > > > > > > > > Chunjun is a distributed ETL tool and distributed data > >> > > > integration > >> > > > > > > tool. Currently Chunjun is based on Apache Flink. It was > >> > initially > >> > > > > known as > >> > > > > > > FlinkX and renamed Chunjun on February 22, 2022. > >> > > > > > > > > - Chunjun codebase: https://github.com/DTStack/chunjun > >> > > > > > > > > > >> > > > > > > > > ##Proposal > >> > > > > > > > > We propose to contribute the Chunjun codebase to the > >> > Apache > >> > > > > Software > >> > > > > > > Foundation with the intent of forming a productive, > >> > meritocratic > >> > > and > >> > > > > open > >> > > > > > > community around Chunjun’s continued development, > >> > according to the > >> > > > > 'Apache > >> > > > > > > Way'. The Chunjun's source code is already under the > >> > Apache License > >> > > > > Version > >> > > > > > > 2.0. > >> > > > > > > > > > >> > > > > > > > > ##Background > >> > > > > > > > > We developed Chunjun in DTStack company in 2017, when > >> > we > >> > > needed a > >> > > > > > > low-code development and high-performance data integration > >> > tool. It > >> > > > has > >> > > > > > > been an open-source project on GitHub since April 2018. > >> > Chunjun is > >> > > > > running > >> > > > > > > in DTStack production environment all the time. Chunjun > >> > has also > >> > > been > >> > > > > > > widely used by companies in China, including DTStack ( > >> > > > > > > https://www.dtstack.com/), Qihu360(https://www.360.cn/), > >> > Iflytek ( > >> > > > > > > https://www.iflytek.com/), XPeng Motors > >> > (https://en.xiaopeng.com/ > >> > > ), > >> > > > > > > WeBank (https://www.webank.com/), > >> > Asiainfo(https://asiainfo.com/), > >> > > > > Guazi( > >> > > > > > > https://www.guazi.com/), Hello Inc > >> > (https://www.hello-inc.com/), > >> > > > etc. > >> > > > > > > Nowadays, Chunjun has a strong community in China. > >> > > > > > > > > > >> > > > > > > > > ##Rationale > >> > > > > > > > > High-performance of Chunjun is based on Apache Flink, > >> > and > >> > > Chunjun > >> > > > > can > >> > > > > > > integrate data from different data source. Users only need > >> > to > >> > > > > configure a > >> > > > > > > JSON file to complete the data reading, transformer, and > >> > writing. > >> > > > > Users can > >> > > > > > > implement new reader/writer plugins to meet their > >> > requirements. > >> > > > Chunjun > >> > > > > > > have implemented plugins that can capture data change for > >> > MySQL to > >> > > > > restore > >> > > > > > > data for Apache Doris. > >> > > > > > > > > Chunjun has the following feature: > >> > > > > > > > > real-time and offline integrate data from different > >> > data > >> > > sources. > >> > > > > > > > > change data capture(CDC) to merge restore data. > >> > > > > > > > > resume from broken-point. > >> > > > > > > > > capture and collect dirty data. > >> > > > > > > > > limit data transferring rate. > >> > > > > > > > > thoughput metrics. > >> > > > > > > > > capture and restore schema evolution. (TODO) > >> > > > > > > > > > >> > > > > > > > > ##Current Status###Meritocracy > >> > > > > > > > > Since Chunjun was open-sourced, many enterprises have > >> > adopted > >> > > > > Chunjun > >> > > > > > > to build up their data integration system. In return, we > >> > have > >> > > > received > >> > > > > many > >> > > > > > > issue reports or enhancements from them simultaneously. The > >> > > codebase > >> > > > > is now > >> > > > > > > mainly managed by the development team inside DTStack who's > >> > > > > responsible for > >> > > > > > > building internal data integration system too.###Community > >> > > > > > > > > Chunjun has been building a community around > >> > contributors and > >> > > > > users to > >> > > > > > > this framework for the last five years. We organized one > >> > meetup in > >> > > > > 2020. > >> > > > > > > Currently, we communicate in Github issues and in chinese > >> > DingTalk > >> > > > > group. > >> > > > > > > There are about 3000 people in this group. And we believe > >> > that we > >> > > can > >> > > > > get a > >> > > > > > > lot of help from the Apache Flink community too. We will > >> > organize a > >> > > > > meetup > >> > > > > > > again in 2022.###Core Developers > >> > > > > > > > > (In alphabetical order) > >> > > > > > > > > Chao Xu (https://github.com/zoudaokoulife) > >> > > > > > > > > Gongjiang Tang, (https://github.com/kyo-tom) > >> > > > > > > > > Huai Yang, (https://github.com/yanghuaiGit) > >> > > > > > > > > Jiangbo Li, (https://github.com/lijiangbo) > >> > > > > > > > > Luning Wong, (https://github.com/deadwind4) > >> > > > > > > > > Luo Li, (https://github.com/kanata163) > >> > > > > > > > > Sishu Yang, (https://github.com/yangsishu) > >> > > > > > > > > Tianzhu Wen, (https://github.com/WTZ468071157) > >> > > > > > > > > Weiliang Hao, (https://github.com/xiuzhu9527) > >> > > > > > > > > Wenqiang Liu, (https://github.com/meng1222) > >> > > > > > > > > Xing Liu, (https://github.com/simenliuxing) > >> > > > > > > > > Yang Lan, (https://github.com/HiLany) > >> > > > > > > > > Yanquan Lv, (https://github.com/lvyanquan) > >> > > > > > > > > Yifan Hu, (https://github.com/demotto) > >> > > > > > > > > Zaiyue Yu, (https://github.com/tonybobam) > >> > > > > > > > > Zhangwan Zhao, (https://github.com/jiemotongxue) > >> > > > > > > > > Zhiqiang Li, (https://github.com/ChestnutQiang) > >> > > > > > > > > They are almost working in real-time computing engine > >> > team of > >> > > > > DTStack. > >> > > > > > > Only Yifan Hu working for CaoCao Tech. Most of them are > >> > Apache > >> > > Flink > >> > > > > > > contributor. > >> > > > > > > > > > >> > > > > > > > > ##Known Risks###Project Name > >> > > > > > > > > The name of the project is Chunjun. Chunjun comes from > >> > mandarin > >> > > > > > > chinese Pinyin "Chun Jun", and it is one of the top ten > >> > famous > >> > > swords > >> > > > > in > >> > > > > > > China.###Orphaned products > >> > > > > > > > > More than 20 contributors and thousands of forks and > >> > star > >> > > further > >> > > > > show > >> > > > > > > that Chunjun is actively supported, and we seek to further > >> > prosper > >> > > > the > >> > > > > > > community with the aid of Apache. As a consequence, > >> > Chunjun is > >> > > > > unlikely to > >> > > > > > > be reduced to an orphaned project.###Inexperience with > >> > Open Source > >> > > > > > > > > Many of the Chunjun committers have experience working > >> > on open > >> > > > > source > >> > > > > > > projects. They are also active contributors to other Apache > >> > > projects. > >> > > > > > > > > ###Homogenous Developers > >> > > > > > > > > The most of core developers are from DTStack, and > >> > Chunjun > >> > > > received > >> > > > > > > some bug fixes and enhancements from other developers not > >> > working > >> > > at > >> > > > > > > DTStack. ###Reliance on Salaried Developers > >> > > > > > > > > Currently, most of core developers are paid to work on > >> > Chunjun > >> > > > > project > >> > > > > > > by DTStack. We look forward to attracting more people > >> > outside > >> > > DTStack > >> > > > > to > >> > > > > > > join this project.###Relationships with Other Apache > >> > Products > >> > > > > > > > > We have integrated with Apache Flink, Apache Hadoop, > >> > Apache > >> > > > Common > >> > > > > and > >> > > > > > > Apache HttpComponents, Log4J and Maven. > >> > > > > > > > > Usage of Apache projects related to Chunjun plugin > >> > > > > > > > > Apache Hive > >> > > > > > > > > Apache Solr > >> > > > > > > > > Apache Doris > >> > > > > > > > > Apache HBase > >> > > > > > > > > Apache Kudu > >> > > > > > > > > Apache Kafka > >> > > > > > > > > Apache Pulsar (TODO)###An Excessive Fascination with > >> > the Apache > >> > > > > Brand > >> > > > > > > > > We acknowledge the value and reputation that the > >> > Apache brand > >> > > > would > >> > > > > > > bring to Chunjun. However, our primary interest is in the > >> > excellent > >> > > > > > > community provided by Apache Software Foundation, in which > >> > all the > >> > > > > projects > >> > > > > > > could gain stability for long-term development. > >> > > > > > > > > > >> > > > > > > > > ##Documentation > >> > > > > > > > > A complete set of documents is provided on GitHub, > >> > including > >> > > > > English > >> > > > > > > and Simplified Chinese versions. > >> > > > > > > > > English: > >> > > > https://github.com/DTStack/chunjun/blob/master/README.md > >> > > > > > > > > Chinese: > >> > > > > https://github.com/DTStack/chunjun/blob/master/README_CH.md > >> > > > > > > > > > >> > > > > > > > > ##Initial Code > >> > > > > > > > > https://github.com/DTStack/chunjun > >> > > > > > > > > > >> > > > > > > > > ##Initial Source and Intellectual Property Submission > >> > Plan > >> > > > > > > > > The codebase is already licensed under the Apache > >> > License 2.0 > >> > > and > >> > > > > the > >> > > > > > > copyright is assigned to DTStack. If the project enters > >> > incubator, > >> > > > > DTStack > >> > > > > > > will transfer the source code & trademark ownership to ASF > >> > via a > >> > > > > Software > >> > > > > > > Grant Agreement. Our initial committers will submit > >> > iCLA(s), SGA, > >> > > and > >> > > > > > > CCLA(s). ##External DependenciesApache-2.0 licenses > >> > > > > > > > > Apache Avro > >> > > > > > > > > Apache Commons > >> > > > > > > > > Apache Curator > >> > > > > > > > > Apache Flink > >> > > > > > > > > Apache Hadoop > >> > > > > > > > > Apache HttpComponents > >> > > > > > > > > Apache Log4j > >> > > > > > > > > Gson > >> > > > > > > > > Guava > >> > > > > > > > > Jackson > >> > > > > > > > > Powermock > >> > > > > > > > > PrometheusEclipse Distribution License > >> > > > > > > > > JUnitEPL licenses > >> > > > > > > > > LogbackMIT licenses > >> > > > > > > > > Mockito > >> > > > > > > > > SLF4J > >> > > > > > > > > > >> > > > > > > > > ##Required Resources ###Git Repositories > >> > > > > > > > > https://github.com/apache/incubator-chunjun###Issue > >> > Tracking > >> > > > > > > > > The community would like to continue using GitHub > >> > > > Issues.###Mailing > >> > > > > > > List > >> > > > > > > > > priv...@chunjun.incubator.apache.org > >> > > > > > > > > d...@chunjun.incubator.apache.org > >> > > > > > > > > comm...@chunjun.incubator.apache.org###Continuous > >> > > <http://comm...@chunjun.incubator.apache.org#%23%23Continuous> > >> > > > <http://comm...@chunjun.incubator.apache.org#%23%23Continuous> > >> > > > > <http://comm...@chunjun.incubator.apache.org#%23%23Continuous> > >> > > > > > > > >> > <http://comm...@chunjun.incubator.apache.org#%23%23Continuous> > >> > > > > > > Integration tool > >> > > > > > > > > GitHub Action > >> > > > > > > > > > >> > > > > > > > > ##Initial Committers > >> > > > > > > > > (In alphabetical order) > >> > > > > > > > > Chao Xu (https://github.com/zoudaokoulife, xuchao at > >> > dtstack > >> > > dot > >> > > > > com) > >> > > > > > > > > Luning Wong (https://github.com/deadwind4, gfeng48 at > >> > gmail > >> > > dot > >> > > > > com) > >> > > > > > > > > Sishu Yang (https://github.com/yangsishu, sishu at > >> > dtstack dot > >> > > > > com) > >> > > > > > > > > Yang Huai (https://github.com/yanghuaiGit, dujie at > >> > dtstack > >> > > dot > >> > > > > com) > >> > > > > > > > > Zhiqiang Li (https://github.com/ChestnutQiang, wujuan > >> > at > >> > > dtstack > >> > > > > dot > >> > > > > > > com) > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > ##Affiliations > >> > > > > > > > > The initial committers are employees of DTStack. The > >> > nominated > >> > > > > mentors > >> > > > > > > and champion are employees of TODO. > >> > > > > > > > > > >> > > > > > > > > ##Sponsors > >> > > > > > > > > > >> > > > > > > > > ###Champion > >> > > > > > > > > TODO ###Nominated Mentors > >> > > > > > > > > TODO > >> > > > > > > > >> > > > > > > > >> > > > >> > --------------------------------------------------------------------- > >> > > > > > > To unsubscribe, e-mail: > >> > general-unsubscr...@incubator.apache.org > >> > > > > > > For additional commands, e-mail: > >> > general-h...@incubator.apache.org > >> > > > > > > > >> > > > > > > > >> > > > > > >> > > > > > >> > --------------------------------------------------------------------- > >> > > > > To unsubscribe, e-mail: > >> > general-unsubscr...@incubator.apache.org > >> > > > > For additional commands, e-mail: > >> > general-h...@incubator.apache.org > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > -- > >> > > Best wishes！ > >> > > CalvinKirs > >> > > > >> > > > >> > > >> > > >> > > >> > --------------------------------------------------------------------- > >> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > >> > For additional commands, e-mail: general-h...@incubator.apache.org > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
