Awesome to see CloudDataFlow coming to Apache. The Stream Processing area has been in general fragmented with a variety of solutions, hoping the community galvanizes around Apache Data Flow.
We are still in the "Apache Storm" world, Any chance for folks building a "Storm Runner²? On 1/20/16, 9:39 AM, "James Malone" <jamesmal...@google.com.INVALID> wrote: >> Great proposal. I like that your proposal includes a well presented >> roadmap, but I don't see any goals that directly address building a >>larger >> community. Y'all have any ideas around outreach that will help with >> adoption? >> > >Thank you and fair point. We have a few additional ideas which we can put >into the Community section. > > >> >> As a start, I recommend y'all add a section to the proposal on the wiki >> page for "Additional Interested Contributors" so that folks who want to >> sign up to participate in the project can do so without requesting >> additions to the initial committer list. >> >> >This is a great idea and I think it makes a lot of sense to add an >"Additional >Interested Contributors" section to the proposal. > > >> On Wed, Jan 20, 2016 at 10:32 AM, James Malone < >> jamesmal...@google.com.invalid> wrote: >> >> > Hello everyone, >> > >> > Attached to this message is a proposed new project - Apache Dataflow, >>a >> > unified programming model for data processing and integration. >> > >> > The text of the proposal is included below. Additionally, the >>proposal is >> > in draft form on the wiki where we will make any required changes: >> > >> > https://wiki.apache.org/incubator/DataflowProposal >> > >> > We look forward to your feedback and input. >> > >> > Best, >> > >> > James >> > >> > ---- >> > >> > = Apache Dataflow = >> > >> > == Abstract == >> > >> > Dataflow is an open source, unified model and set of language-specific >> SDKs >> > for defining and executing data processing workflows, and also data >> > ingestion and integration flows, supporting Enterprise Integration >> Patterns >> > (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines >>simplify >> > the mechanics of large-scale batch and streaming data processing and >>can >> > run on a number of runtimes like Apache Flink, Apache Spark, and >>Google >> > Cloud Dataflow (a cloud service). Dataflow also brings DSL in >>different >> > languages, allowing users to easily implement their data integration >> > processes. >> > >> > == Proposal == >> > >> > Dataflow is a simple, flexible, and powerful system for distributed >>data >> > processing at any scale. Dataflow provides a unified programming >>model, a >> > software development kit to define and construct data processing >> pipelines, >> > and runners to execute Dataflow pipelines in several runtime engines, >> like >> > Apache Spark, Apache Flink, or Google Cloud Dataflow. Dataflow can be >> used >> > for a variety of streaming or batch data processing goals including >>ETL, >> > stream analysis, and aggregate computation. The underlying programming >> > model for Dataflow provides MapReduce-like parallelism, combined with >> > support for powerful data windowing, and fine-grained correctness >> control. >> > >> > == Background == >> > >> > Dataflow started as a set of Google projects focused on making data >> > processing easier, faster, and less costly. The Dataflow model is a >> > successor to MapReduce, FlumeJava, and Millwheel inside Google and is >> > focused on providing a unified solution for batch and stream >>processing. >> > These projects on which Dataflow is based have been published in >>several >> > papers made available to the public: >> > >> > * MapReduce - http://research.google.com/archive/mapreduce.html >> > >> > * Dataflow model - http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf >> > >> > * FlumeJava - http://notes.stephenholiday.com/FlumeJava.pdf >> > >> > * MillWheel - http://research.google.com/pubs/pub41378.html >> > >> > Dataflow was designed from the start to provide a portable programming >> > layer. When you define a data processing pipeline with the Dataflow >> model, >> > you are creating a job which is capable of being processed by any >>number >> of >> > Dataflow processing engines. Several engines have been developed to >>run >> > Dataflow pipelines in other open source runtimes, including a Dataflow >> > runner for Apache Flink and Apache Spark. There is also a ³direct >> runner², >> > for execution on the developer machine (mainly for dev/debug >>purposes). >> > Another runner allows a Dataflow program to run on a managed service, >> > Google Cloud Dataflow, in Google Cloud Platform. The Dataflow Java >>SDK is >> > already available on GitHub, and independent from the Google Cloud >> Dataflow >> > service. Another Python SDK is currently in active development. >> > >> > In this proposal, the Dataflow SDKs, model, and a set of runners will >>be >> > submitted as an OSS project under the ASF. The runners which are a >>part >> of >> > this proposal include those for Spark (from Cloudera), Flink (from >>data >> > Artisans), and local development (from Google); the Google Cloud >>Dataflow >> > service runner is not included in this proposal. Further references to >> > Dataflow will refer to the Dataflow model, SDKs, and runners which >>are a >> > part of this proposal (Apache Dataflow) only. The initial submission >>will >> > contain the already-released Java SDK; Google intends to submit the >> Python >> > SDK later in the incubation process. The Google Cloud Dataflow service >> will >> > continue to be one of many runners for Dataflow, built on Google Cloud >> > Platform, to run Dataflow pipelines. Necessarily, Cloud Dataflow will >> > develop against the Apache project additions, updates, and changes. >> Google >> > Cloud Dataflow will become one user of Apache Dataflow and will >> participate >> > in the project openly and publicly. >> > >> > The Dataflow programming model has been designed with simplicity, >> > scalability, and speed as key tenants. In the Dataflow model, you only >> need >> > to think about four top-level concepts when constructing your data >> > processing job: >> > >> > * Pipelines - The data processing job made of a series of computations >> > including input, processing, and output >> > >> > * PCollections - Bounded (or unbounded) datasets which represent the >> input, >> > intermediate and output data in pipelines >> > >> > * PTransforms - A data processing step in a pipeline in which one or >>more >> > PCollections are an input and output >> > >> > * I/O Sources and Sinks - APIs for reading and writing data which are >>the >> > roots and endpoints of the pipeline >> > >> > == Rationale == >> > >> > With Dataflow, Google intended to develop a framework which allowed >> > developers to be maximally productive in defining the processing, and >> then >> > be able to execute the program at various levels of >> > latency/cost/completeness without re-architecting or re-writing it. >>This >> > goal was informed by Google¹s past experience developing several >>models, >> > frameworks, and tools useful for large-scale and distributed data >> > processing. While Google has previously published papers describing >>some >> of >> > its technologies, Google decided to take a different approach with >> > Dataflow. Google open-sourced the SDK and model alongside >> commercialization >> > of the idea and ahead of publishing papers on the topic. As a result, >>a >> > number of open source runtimes exist for Dataflow, such as the Apache >> Flink >> > and Apache Spark runners. >> > >> > We believe that submitting Dataflow as an Apache project will provide >>an >> > immediate, worthwhile, and substantial contribution to the open source >> > community. As an incubating project, we believe Dataflow will have a >> better >> > opportunity to provide a meaningful contribution to OSS and also >> integrate >> > with other Apache projects. >> > >> > In the long term, we believe Dataflow can be a powerful abstraction >>layer >> > for data processing. By providing an abstraction layer for data >>pipelines >> > and processing, data workflows can be increasingly portable, >>resilient to >> > breaking changes in tooling, and compatible across many execution >> engines, >> > runtimes, and open source projects. >> > >> > == Initial Goals == >> > >> > We are breaking our initial goals into immediate (< 2 months), >>short-term >> > (2-4 months), and intermediate-term (> 4 months). >> > >> > Our immediate goals include the following: >> > >> > * Plan for reconciling the Dataflow Java SDK and various runners into >>one >> > project >> > >> > * Plan for refactoring the existing Java SDK for better extensibility >>by >> > SDK and runner writers >> > >> > * Validating all dependencies are ASL 2.0 or compatible >> > >> > * Understanding and adapting to the Apache development process >> > >> > Our short-term goals include: >> > >> > * Moving the newly-merged lists, and build utilities to Apache >> > >> > * Start refactoring codebase and move code to Apache Git repo >> > >> > * Continue development of new features, functions, and fixes in the >> > Dataflow Java SDK, and Dataflow runners >> > >> > * Cleaning up the Dataflow SDK sources and crafting a roadmap and plan >> for >> > how to include new major ideas, modules, and runtimes >> > >> > * Establishment of easy and clear build/test framework for Dataflow >>and >> > associated runtimes; creation of testing, rollback, and validation >>policy >> > >> > * Analysis and design for work needed to make Dataflow a better data >> > processing abstraction layer for multiple open source frameworks and >> > environments >> > >> > Finally, we have a number of intermediate-term goals: >> > >> > * Roadmapping, planning, and execution of integrations with other OSS >>and >> > non-OSS projects/products >> > >> > * Inclusion of additional SDK for Python, which is under active >> development >> > >> > == Current Status == >> > >> > === Meritocracy === >> > >> > Dataflow was initially developed based on ideas from many employees >> within >> > Google. As an ASL OSS project on GitHub, the Dataflow SDK has received >> > contributions from data Artisans, Cloudera Labs, and other individual >> > developers. As a project under incubation, we are committed to >>expanding >> > our effort to build an environment which supports a meritocracy. We >>are >> > focused on engaging the community and other related projects for >>support >> > and contributions. Moreover, we are committed to ensure contributors >>and >> > committers to Dataflow come from a broad mix of organizations through >>a >> > merit-based decision process during incubation. We believe strongly in >> the >> > Dataflow model and are committed to growing an inclusive community of >> > Dataflow contributors. >> > >> > === Community === >> > >> > The core of the Dataflow Java SDK has been developed by Google for use >> with >> > Google Cloud Dataflow. Google has active community engagement in the >>SDK >> > GitHub repository ( >> https://github.com/GoogleCloudPlatform/DataflowJavaSDK >> > ), >> > on Stack Overflow ( >> > http://stackoverflow.com/questions/tagged/google-cloud-dataflow) and >>has >> > had contributions from a number of organizations and indivuduals. >> > >> > Everyday, Cloud Dataflow is actively used by a number of organizations >> and >> > institutions for batch and stream processing of data. We believe >> acceptance >> > will allow us to consolidate existing Dataflow-related work, grow the >> > Dataflow community, and deepen connections between Dataflow and other >> open >> > source projects. >> > >> > === Core Developers === >> > >> > The core developers for Dataflow and the Dataflow runners are: >> > >> > * Frances Perry >> > >> > * Tyler Akidau >> > >> > * Davor Bonaci >> > >> > * Luke Cwik >> > >> > * Ben Chambers >> > >> > * Kenn Knowles >> > >> > * Dan Halperin >> > >> > * Daniel Mills >> > >> > * Mark Shields >> > >> > * Craig Chambers >> > >> > * Maximilian Michels >> > >> > * Tom White >> > >> > * Josh Wills >> > >> > === Alignment === >> > >> > The Dataflow SDK can be used to create Dataflow pipelines which can be >> > executed on Apache Spark or Apache Flink. Dataflow is also related to >> other >> > Apache projects, such as Apache Crunch. We plan on expanding >> functionality >> > for Dataflow runners, support for additional domain specific >>languages, >> and >> > increased portability so Dataflow is a powerful abstraction layer for >> data >> > processing. >> > >> > == Known Risks == >> > >> > === Orphaned Products === >> > >> > The Dataflow SDK is presently used by several organizations, from >>small >> > startups to Fortune 100 companies, to construct production pipelines >> which >> > are executed in Google Cloud Dataflow. Google has a long-term >>commitment >> to >> > advance the Dataflow SDK; moreover, Dataflow is seeing increasing >> interest, >> > development, and adoption from organizations outside of Google. >> > >> > === Inexperience with Open Source === >> > >> > Google believes strongly in open source and the exchange of >>information >> to >> > advance new ideas and work. Examples of this commitment are active OSS >> > projects such as Chromium (https://www.chromium.org) and Kubernetes ( >> > http://kubernetes.io/). With Dataflow, we have tried to be >>increasingly >> > open and forward-looking; we have published a paper in the VLDB >> conference >> > describing the Dataflow model ( >> > http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf) and were quick to >> release >> > the Dataflow SDK as open source software with the launch of Cloud >> Dataflow. >> > Our submission to the Apache Software Foundation is a logical >>extension >> of >> > our commitment to open source software. >> > >> > === Homogeneous Developers === >> > >> > The majority of committers in this proposal belong to Google due to >>the >> > fact that Dataflow has emerged from several internal Google projects. >> This >> > proposal also includes committers outside of Google who are actively >> > involved with other Apache projects, such as Hadoop, Flink, and Spark. >> We >> > expect our entry into incubation will allow us to expand the number of >> > individuals and organizations participating in Dataflow development. >> > Additionally, separation of the Dataflow SDK from Google Cloud >>Dataflow >> > allows us to focus on the open source SDK and model and do what is >>best >> for >> > this project. >> > >> > === Reliance on Salaried Developers === >> > >> > The Dataflow SDK and Dataflow runners have been developed primarily by >> > salaried developers supporting the Google Cloud Dataflow project. >>While >> the >> > Dataflow SDK and Cloud Dataflow have been developed by different teams >> (and >> > this proposal would reinforce that separation) we expect our initial >>set >> of >> > developers will still primarily be salaried. Contribution has not been >> > exclusively from salaried developers, however. For example, the >>contrib >> > directory of the Dataflow SDK ( >> > >> >>https://github.com/GoogleCloudPlatform/DataflowJavaSDK/tree/master/contri >>b >> > ) >> > contains items from free-time contributors. Moreover, seperate >>projects, >> > such as ScalaFlow (https://github.com/darkjh/scalaflow) have been >> created >> > around the Dataflow model and SDK. We expect our reliance on salaried >> > developers will decrease over time during incubation. >> > >> > === Relationship with other Apache products === >> > >> > Dataflow directly interoperates with or utilizes several existing >>Apache >> > projects. >> > >> > * Build >> > >> > ** Apache Maven >> > >> > * Data I/O, Libraries >> > >> > ** Apache Avro >> > >> > ** Apache Commons >> > >> > * Dataflow runners >> > >> > ** Apache Flink >> > >> > ** Apache Spark >> > >> > Dataflow when used in batch mode shares similarities with Apache >>Crunch; >> > however, Dataflow is focused on a model, SDK, and abstraction layer >> beyond >> > Spark and Hadoop (MapReduce.) One key goal of Dataflow is to provide >>an >> > intermediate abstraction layer which can easily be implemented and >> utilized >> > across several different processing frameworks. >> > >> > === An excessive fascination with the Apache brand === >> > >> > With this proposal we are not seeking attention or publicity. Rather, >>we >> > firmly believe in the Dataflow model, SDK, and the ability to make >> Dataflow >> > a powerful yet simple framework for data processing. While the >>Dataflow >> SDK >> > and model have been open source, we believe putting code on GitHub can >> only >> > go so far. We see the Apache community, processes, and mission as >> critical >> > for ensuring the Dataflow SDK and model are truly community-driven, >> > positively impactful, and innovative open source software. While >>Google >> has >> > taken a number of steps to advance its various open source projects, >>we >> > believe Dataflow is a great fit for the Apache Software Foundation >>due to >> > its focus on data processing and its relationships to existing ASF >> > projects. >> > >> > == Documentation == >> > >> > The following documentation is relevant to this proposal. Relevant >> portion >> > of the documentation will be contributed to the Apache Dataflow >>project. >> > >> > * Dataflow website: https://cloud.google.com/dataflow >> > >> > * Dataflow programming model: >> > https://cloud.google.com/dataflow/model/programming-model >> > >> > * Codebases >> > >> > ** Dataflow Java SDK: >> > https://github.com/GoogleCloudPlatform/DataflowJavaSDK >> > >> > ** Flink Dataflow runner: >>https://github.com/dataArtisans/flink-dataflow >> > >> > ** Spark Dataflow runner: https://github.com/cloudera/spark-dataflow >> > >> > * Dataflow Java SDK issue tracker: >> > https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues >> > >> > * google-cloud-dataflow tag on Stack Overflow: >> > http://stackoverflow.com/questions/tagged/google-cloud-dataflow >> > >> > == Initial Source == >> > >> > The initial source for Dataflow which we will submit to the Apache >> > Foundation will include several related projects which are currently >> hosted >> > on the GitHub repositories: >> > >> > * Dataflow Java SDK ( >> > https://github.com/GoogleCloudPlatform/DataflowJavaSDK) >> > >> > * Flink Dataflow runner >>(https://github.com/dataArtisans/flink-dataflow) >> > >> > * Spark Dataflow runner (https://github.com/cloudera/spark-dataflow) >> > >> > These projects have always been Apache 2.0 licensed. We intend to >>bundle >> > all of these repositories since they are all complimentary and should >>be >> > maintained in one project. Prior to our submission, we will combine >>all >> of >> > these projects into a new git repository. >> > >> > == Source and Intellectual Property Submission Plan == >> > >> > The source for the Dataflow SDK and the three runners (Spark, Flink, >> Google >> > Cloud Dataflow) are already licensed under an Apache 2 license. >> > >> > * Dataflow SDK - >> > >> >>https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/LICENS >>E >> > >> > * Flink runner - >> > https://github.com/dataArtisans/flink-dataflow/blob/master/LICENSE >> > >> > * Spark runner - >> > https://github.com/cloudera/spark-dataflow/blob/master/LICENSE >> > >> > Contributors to the Dataflow SDK have also signed the Google >>Individual >> > Contributor License Agreement ( >> > https://cla.developers.google.com/about/google-individual) in order to >> > contribute to the project. >> > >> > With respect to trademark rights, Google does not hold a trademark on >>the >> > phrase ³Dataflow.² Based on feedback and guidance we receive during >>the >> > incubation process, we are open to renaming the project if necessary >>for >> > trademark or other concerns. >> > >> > == External Dependencies == >> > >> > All external dependencies are licensed under an Apache 2.0 or >> > Apache-compatible license. As we grow the Dataflow community we will >> > configure our build process to require and validate all contributions >>and >> > dependencies are licensed under the Apache 2.0 license or are under an >> > Apache-compatible license. >> > >> > == Required Resources == >> > >> > === Mailing Lists === >> > >> > We currently use a mix of mailing lists. We will migrate our existing >> > mailing lists to the following: >> > >> > * d...@dataflow.incubator.apache.org >> > >> > * u...@dataflow.incubator.apache.org >> > >> > * priv...@dataflow.incubator.apache.org >> > >> > * comm...@dataflow.incubator.apache.org >> > >> > === Source Control === >> > >> > The Dataflow team currently uses Git and would like to continue to do >>so. >> > We request a Git repository for Dataflow with mirroring to GitHub >> enabled. >> > >> > === Issue Tracking === >> > >> > We request the creation of an Apache-hosted JIRA. The Dataflow >>project is >> > currently using both a public GitHub issue tracker and internal Google >> > issue tracking. We will migrate and combine from these two sources to >>the >> > Apache JIRA. >> > >> > == Initial Committers == >> > >> > * Aljoscha Krettek [aljos...@apache.org] >> > >> > * Amit Sela [amitsel...@gmail.com] >> > >> > * Ben Chambers [bchamb...@google.com] >> > >> > * Craig Chambers [chamb...@google.com] >> > >> > * Dan Halperin [dhalp...@google.com] >> > >> > * Davor Bonaci [da...@google.com] >> > >> > * Frances Perry [f...@google.com] >> > >> > * James Malone [jamesmal...@google.com] >> > >> > * Jean-Baptiste Onofré [jbono...@apache.org] >> > >> > * Josh Wills [jwi...@apache.org] >> > >> > * Kostas Tzoumas [kos...@data-artisans.com] >> > >> > * Kenneth Knowles [k...@google.com] >> > >> > * Luke Cwik [lc...@google.com] >> > >> > * Maximilian Michels [m...@apache.org] >> > >> > * Stephan Ewen [step...@data-artisans.com] >> > >> > * Tom White [t...@cloudera.com] >> > >> > * Tyler Akidau [taki...@google.com] >> > >> > == Affiliations == >> > >> > The initial committers are from six organizations. Google developed >> > Dataflow and the Dataflow SDK, data Artisans developed the Flink >>runner, >> > and Cloudera (Labs) developed the Spark runner. >> > >> > * Cloudera >> > >> > ** Tom White >> > >> > * Data Artisans >> > >> > ** Aljoscha Krettek >> > >> > ** Kostas Tzoumas >> > >> > ** Maximilian Michels >> > >> > ** Stephan Ewen >> > >> > * Google >> > >> > ** Ben Chambers >> > >> > ** Dan Halperin >> > >> > ** Davor Bonaci >> > >> > ** Frances Perry >> > >> > ** James Malone >> > >> > ** Kenneth Knowles >> > >> > ** Luke Cwik >> > >> > ** Tyler Akidau >> > >> > * PayPal >> > >> > ** Amit Sela >> > >> > * Slack >> > >> > ** Josh Wills >> > >> > * Talend >> > >> > ** Jean-Baptiste Onofré >> > >> > == Sponsors == >> > >> > === Champion === >> > >> > * Jean-Baptiste Onofre [jbono...@apache.org] >> > >> > === Nominated Mentors === >> > >> > * Jim Jagielski [j...@apache.org] >> > >> > * Venkatesh Seetharam [venkat...@apache.org] >> > >> > * Bertrand Delacretaz [bdelacre...@apache.org] >> > >> > * Ted Dunning [tdunn...@apache.org] >> > >> > === Sponsoring Entity === >> > >> > The Apache Incubator >> > >> >> >> >> -- >> Sean >> --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
