Thanks for the clarification Bill! -Taylor
> On Jun 15, 2017, at 1:24 PM, Bill Graham <billgra...@gmail.com> wrote: > > Hi Taylor, > > The Heron team engaged with members of the Apache Storm community through > private channels before the project was made available as open source. We > recognize this is not the ideal approach and going forward we will use more > collaborative methods as we progress and grow the Heron community. > > One of our goals during incubation will be to use open forums of > communication, like the Apache mailing lists, and work to foster a truly > collaborative environment for both Apache Storm and Heron community members > to work within together. > > The Fabric team at Google uses Heron extensively. > > thanks, > Bill > > On Thu, Jun 15, 2017 at 10:42 AM, Debo Dutta (dedutta) <dedu...@cisco.com> > wrote: > >> Am happy to help too! >> >> Thx >> Debo >> >> Sent from my iPhone >> >>> On Jun 14, 2017, at 8:05 PM, William Markito Oliveira < >> william.mark...@gmail.com> wrote: >>> >>> Howdy! >>> >>> If Heron is looking for some help around incubation process, I'd love to >>> help while Geode experience is still fresh in my mind and given that >> it's a >>> project/space that I do have interest. Since I'm not an ASF member, I >> don't >>> think I can offer to be a mentor, but can probably still help and >>> participate on the process. >>> >>> Thanks! >>> >>>> On Wed, Jun 14, 2017 at 7:54 PM, P. Taylor Goetz <ptgo...@gmail.com> >> wrote: >>>> >>>> Hi Bill/Supun, >>>> >>>> Sorry for not being a little more clear. I was asking more about how the >>>> Heron community would seek to engage with Storm community at the >>>> *community* level as opposed to the technical level (i.e. “Community >> over >>>> Code”). >>>> >>>> I’ve been asked by many why this has never happened, and have always >>>> struggled to answer. Maybe you could help answer that question as well >> as >>>> if and how that might change if Heron were to incubate. >>>> >>>> Another quick question: The proposal mentions Heron being used in >>>> production at Google, but some Google employees I recently spoke to >> seemed >>>> to contradict that. Could you explain? Note that’s nothing that would >>>> preclude the project from incubating, I’m just curious. >>>> >>>> -Taylor >>>> >>>>> On Jun 14, 2017, at 7:35 AM, Supun Kamburugamuve <supu...@gmail.com> >>>> wrote: >>>>> >>>>> Hi Taylor, >>>>> >>>>> For me, one of the interesting differences between Heron and Storm is >> the >>>>> execution model. Storm uses a shared memory model while Heron uses a >>>>> process based model. It will be interesting to see how these two >> evolve. >>>>> >>>>> Thanks, >>>>> Supun.. >>>>> >>>>> On Mon, Jun 12, 2017 at 4:15 PM, Bill Graham <billgra...@gmail.com> >>>> wrote: >>>>> >>>>>> Hi Taylor, >>>>>> >>>>>> Thanks for the mentor offer, we'd be glad to have your help. >>>>>> >>>>>> I think the best place for collaboration would be around the evolution >>>> of >>>>>> the API. In addition we plan to look more into DSL solutions which we >>>> could >>>>>> potentially collaborate on. This could be Trident, or Beam or >> something >>>>>> else, but there could be synergies for future development here. >>>>>> >>>>>> thanks, >>>>>> Bill >>>>>> >>>>>> On Fri, Jun 9, 2017 at 8:53 PM, P. Taylor Goetz <ptgo...@gmail.com> >>>> wrote: >>>>>> >>>>>>> Hi Bill, >>>>>>> >>>>>>> Could you comment on how/if the Heron community would be willing to >>>> work >>>>>>> with the Storm community? I've seen a number of new features in Storm >>>>>> being >>>>>>> ported to Heron, but I have yet to see any attempt by the Heron >>>> community >>>>>>> to engage with the Apache Storm community. >>>>>>> >>>>>>> I don't think it would be too far off to say that the relationship >>>>>> between >>>>>>> Heron and Apache Storm has been somewhat adversarial. The pre- and >>>>>>> post-open sourcing marketing around Heron seemed, at least to me, >>>>>> somewhat >>>>>>> aggressively negative toward Storm. >>>>>>> >>>>>>> As a peer to Apache Storm, how would the proposed "Apache Heron" >>>>>> community >>>>>>> work to collaborate with the Storm community? If Heron is adopting >> API >>>>>>> changes in Storm, then it seems there is an opportunity for >>>>>> collaboration. >>>>>>> >>>>>>> Don't take any of this as an objection to incubating the project. I >>>> would >>>>>>> support it. I would also be willing to be a mentor, if you would >>>> consider >>>>>>> taking on another. >>>>>>> >>>>>>> -Taylor >>>>>>> >>>>>>>> On Jun 8, 2017, at 1:23 PM, Bill Graham <billgra...@gmail.com> >> wrote: >>>>>>>> >>>>>>>> Dear Apache Incubator Community, >>>>>>>> >>>>>>>> We are excited to share our proposal for discussion and feedback >>>>>>>> for entering Apache Incubation. Heron is a real-time, distributed, >>>>>>>> fault-tolerant stream processing engine. >>>>>>>> >>>>>>>> Our proposal can be found at https://wiki.apache.org/ >>>>>>> incubator/HeronProposal >>>>>>>> and is included below. >>>>>>>> >>>>>>>> >>>>>>>> Thank you, >>>>>>>> >>>>>>>> Bill Graham on behalf of the Heron developers >>>>>>>> >>>>>>>> >>>>>>>> # Heron Proposal >>>>>>>> >>>>>>>> ## Abstract >>>>>>>> Heron is a real-time, distributed, fault-tolerant stream processing >>>>>>> engine >>>>>>>> initially developed by Twitter. >>>>>>>> >>>>>>>> ## Proposal >>>>>>>> >>>>>>>> Heron is a real-time stream processing engine built for high >>>>>> performance, >>>>>>>> ease of manageability, performance predictability and developer >>>>>>>> productivity[1]. We wish to develop a community around Heron to >>>>>> increase >>>>>>>> contributions and see Heron thrive in an open forum. >>>>>>>> >>>>>>>> ## Background >>>>>>>> >>>>>>>> Heron provides the ability for developers to compose directed >> acyclic >>>>>>>> graphs (DAGs) of real-time query execution logic (i.e. a topology) >> and >>>>>>>> submit the topology to execute on a pluggable job scheduling system >>>>>>> (e.g., >>>>>>>> Apache Aurora, YARN, Marathon, etc). Users can employ either the >>>> native >>>>>>>> Heron API or the Apache Storm API to develop the topology. Heron >>>>>> supports >>>>>>>> the Storm API for ease of migration, but beyond that Heron’s >>>>>> architecture >>>>>>>> differs considerably from Storm’s. >>>>>>>> >>>>>>>> Users submit a topology to the scheduler using the Heron client, >> which >>>>>>> uses >>>>>>>> the Heron binary libraries to deploy all daemons required to run and >>>>>>> manage >>>>>>>> the topology. The topology therefore has no reliance on centrally >>>>>> managed >>>>>>>> Heron services, only on a generic job scheduling system, which lends >>>>>>> itself >>>>>>>> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN >>>>>> (among >>>>>>>> others). >>>>>>>> >>>>>>>> The scheduler runs each topology as a job consisting of multiple >>>>>>>> containers. One of the containers runs the topology master, >>>> responsible >>>>>>> for >>>>>>>> managing the topology. The remaining containers each runs a stream >>>>>>> manager >>>>>>>> responsible for data routing, a metrics manager that collects and >>>>>> reports >>>>>>>> various metrics and a number of processes called Heron instances >> which >>>>>>> run >>>>>>>> the user-defined logic on the stream of tuples. Parallelism is >>>> achieved >>>>>>> via >>>>>>>> process-based isolation of Heron instances, which provides >> predictable >>>>>>>> performance while simplifying debugging. The containers are >> allocated >>>>>> and >>>>>>>> managed by the scheduler framework based on resource availability of >>>>>>> nodes >>>>>>>> in the cluster. The metadata for the topology, such as the physical >>>>>> plan >>>>>>>> and execution details, are stored in the pluggable Heron State >> Manager >>>>>>>> (e.g. Apache ZooKeeper). >>>>>>>> >>>>>>>> ## Rationale >>>>>>>> >>>>>>>> Heron is a general-purpose, modular and extensible platform that can >>>> be >>>>>>>> leveraged to support common, real-time analytics use cases. There is >>>> an >>>>>>>> increasing demand for open-source, scalable real-time analytics >>>>>> systems. >>>>>>> We >>>>>>>> believe that Heron can be leveraged by other organizations to build >>>>>>>> streaming applications that can benefit from its robustness, high >>>>>>>> performance, adaptability to cloud environments and ease of use. >>>>>>> Moreover, >>>>>>>> we hope that open-sourcing Heron will help to further evolve the >>>>>>> technology >>>>>>>> as the project attracts contributors with diverse backgrounds and >>>> areas >>>>>>> of >>>>>>>> expertise. >>>>>>>> >>>>>>>> We believe the Apache foundation is a great fit as the long-term >> home >>>>>> for >>>>>>>> Heron, as it provides an established process for community-driven >>>>>>>> development and decision making by consensus. This is exactly the >>>> model >>>>>>> we >>>>>>>> want for future Heron development. >>>>>>>> >>>>>>>> ## Initial Goals >>>>>>>> >>>>>>>> * Move the existing codebase, website, documentation, and mailing >>>> lists >>>>>>> to >>>>>>>> Apache-hosted infrastructure. >>>>>>>> * Integrate with the Apache development process. >>>>>>>> * Ensure all dependencies are compliant with Apache License version >>>>>> 2.0. >>>>>>>> * Incrementally develop and release per Apache guidelines. >>>>>>>> >>>>>>>> ## Current Status >>>>>>>> >>>>>>>> Heron is a stable project used in production at Twitter since 2014 >> and >>>>>>> open >>>>>>>> sourced under the ASL v2 license in 2016. The Heron source code is >>>>>>>> currently hosted at github.com (https://github.com/twitter/heron), >>>>>> which >>>>>>>> will seed the Apache git repository. >>>>>>>> >>>>>>>> ### Meritocracy >>>>>>>> >>>>>>>> By submitting this incubator proposal, we’re expressing our intent >> to >>>>>>> build >>>>>>>> a diverse developer community around Heron that will conduct itself >>>>>>>> according to The Apache Way and use a meritocratic means of building >>>>>> it's >>>>>>>> committer base. Several companies and universities have already >>>>>> expressed >>>>>>>> interest in and contributed to Heron. Our goal is to grow the Heron >>>>>>>> community by encouraging open communication, contribution and >>>>>>> participation >>>>>>>> of all types, and ensuring that contributors are recognized >>>>>>> appropriately. >>>>>>>> >>>>>>>> ### Community >>>>>>>> >>>>>>>> Heron is currently being used by Twitter, Google, Machine Zone and >>>>>>>> ndustrial.io and has received significant contributions by >> Microsoft >>>>>> and >>>>>>>> Streamlio. By bringing Heron into the Apache ecosystem, we believe >> we >>>>>> can >>>>>>>> attract even more developers who are interested in creating >> real-time >>>>>>>> systems to build the project's contributor base. >>>>>>>> >>>>>>>> ### Core Developers >>>>>>>> >>>>>>>> Current core developers are engineers from Twitter, Google, >> Microsoft >>>>>> and >>>>>>>> Streamlio. >>>>>>>> >>>>>>>> ### Alignment >>>>>>>> >>>>>>>> Heron utilizes a number of Apache technologies. Heron leverages >> Apache >>>>>>>> ZooKeeper for coordination and has scheduler implementations to >>>>>> integrate >>>>>>>> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via >> Apache >>>>>>> REEF) >>>>>>>> as well as spout implementations to integrate with Apache Kafka and >>>>>>> metrics >>>>>>>> implementations to integrate with Scribe. Heron also implements the >>>>>>> Apache >>>>>>>> Storm user-level API, which allows topologies written against Storm >> to >>>>>>> run >>>>>>>> in Heron. We believe that having Heron at Apache will help further >> the >>>>>>>> growth of the streaming compute community, as well as encourage >>>>>>> cooperation >>>>>>>> and developer cross pollination with other Apache projects. >>>>>>>> >>>>>>>> ## Known Risks >>>>>>>> >>>>>>>> ### Orphaned Products >>>>>>>> >>>>>>>> The risk of the Heron project being abandoned is minimal. It is used >>>> in >>>>>>>> production at Twitter and Google and other companies are evaluating >> or >>>>>>>> adopting it for production use. >>>>>>>> >>>>>>>> ### Inexperience with Open Source >>>>>>>> >>>>>>>> All of the core contributors to the project have considerable >>>>>> experience >>>>>>>> with open source software development. Bill Graham[2], Ashvin >>>>>> Agrawal[3] >>>>>>>> and Supun Kamburugamuve[4], committers on the project, are PMCs on >>>>>> other >>>>>>>> Apache projects and Bill and Ashvin have gone through the Apache >>>>>>> incubator >>>>>>>> process. Twitter has already donated numerous projects to the ASF >>>>>> (e.g., >>>>>>>> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be >>>>>> mentored >>>>>>>> by experienced ASF members that can help with any roadblocks. >>>>>>>> >>>>>>>> ### Homogenous Developers >>>>>>>> >>>>>>>> Initial committers come from 5 separate organizations. Our intention >>>> is >>>>>>>> increase the diversity of contributing developers and their >>>>>> affiliations. >>>>>>>> To date github contributions have come from approximately 50 >>>>>> contributors >>>>>>>> from outside the Twitter team. >>>>>>>> >>>>>>>> ### Reliance on Salaried Developers >>>>>>>> >>>>>>>> It is expected that Heron development will occur on both salaried >> time >>>>>>> and >>>>>>>> on volunteer time. The majority of initial committers are paid by >>>> their >>>>>>>> employers to contribute to this project. We are committed to >>>> recruiting >>>>>>>> additional committers from other organizations as well as >> non-salaried >>>>>>>> committers to join project. >>>>>>>> >>>>>>>> ### Relationships with Other Apache Products >>>>>>>> >>>>>>>> As mentioned in the Alignment section, Heron implements the Apache >>>>>> Storm >>>>>>>> API and integrates with multiple Apache schedulers (Apache Mesos, >>>>>> Apache >>>>>>>> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and >>>> Apache >>>>>>>> Thrift. >>>>>>>> >>>>>>>> ### An Excessive Fascination with the Apache Brand >>>>>>>> >>>>>>>> Heron's popularity is growing in the streaming compute space and we >>>> are >>>>>>>> long time supporters of the Apache brand. This proposal is not for >> the >>>>>>>> purpose of generating publicity through. Rather, the primary >> benefits >>>>>> to >>>>>>>> joining Apache are those of community building and open decision >>>> making >>>>>>>> outlined in the Rationale section. >>>>>>>> >>>>>>>> ## Documentation >>>>>>>> >>>>>>>> This proposal exists online as http://wiki.apache.org/ >>>>>>>> incubator/HeronProposal. Extensive documentation can be found on >>>> github >>>>>>> at >>>>>>>> https://twitter.github.io/heron and the source code is well >>>>>> documented. >>>>>>>> >>>>>>>> ## Source and Intellectual Property Submission Plan >>>>>>>> >>>>>>>> The Heron codebase is currently hosted on Github: >>>>>>>> https://github.com/twitter/heron. During incubation, the codebase >>>> will >>>>>>> be >>>>>>>> migrated to Apache infrastructure. The source code is already ASF >> 2.0 >>>>>>>> licensed. >>>>>>>> >>>>>>>> ## External Dependencies >>>>>>>> >>>>>>>> All external libraries have ASF 2.0 compatible licenses except for >>>>>>> pylint. >>>>>>>> The pylint library is GPL licensed, but is only used for pre-build >>>>>> Python >>>>>>>> style checks and is neither bundled with, nor relied upon by, the >>>> Heron >>>>>>>> source or binary release artifacts. >>>>>>>> >>>>>>>> ## Cryptography >>>>>>>> >>>>>>>> Heron does not use any cryptography libraries. >>>>>>>> >>>>>>>> ## Required Resources >>>>>>>> >>>>>>>> ### Mailing lists >>>>>>>> >>>>>>>> priv...@heron.incubator.apache.org (with moderated subscriptions) >>>>>>>> d...@heron.incubator.apache.org >>>>>>>> comm...@heron.incubator.apache.org >>>>>>>> u...@heron.incubator.apache.org >>>>>>>> >>>>>>>> ## Subversion Directory >>>>>>>> >>>>>>>> Git is the preferred source control system: git:// >>>> git.apache.org/heron >>>>>>>> >>>>>>>> ## Issue Tracking >>>>>>>> >>>>>>>> JIRA: Heron (HERON) >>>>>>>> >>>>>>>> ## Initial Committers >>>>>>>> >>>>>>>> * Andrew Jorgensen (andrew at andrewjorgensen dot com) >>>>>>>> * Ashvin Agrawal (ashvin at apache dot org)* >>>>>>>> * Avrilia Floratou (avrilia dot floratou at gmail dot com) >>>>>>>> * Bill Graham (billgraham at apache dot org)* >>>>>>>> * Brian Hatfield (bmhatfield at gmail dot com) >>>>>>>> * Chris Kellogg (cckellogg at gmail dot com) >>>>>>>> * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) >>>>>>>> * Karthik Ramasamy (karthik at gmail dot com) >>>>>>>> * Maosong Fu (maosongfu at gmail dot com) >>>>>>>> * Neng Lu(freeneng at gmail dot com) >>>>>>>> * Runhang Li (obj dot runhang at gmail dot com) >>>>>>>> * Sanjeev Kulkarni (sanjeevrk at gmail dot com) >>>>>>>> * Supun Kamburugamuve (supun at apache dot org)* >>>>>>>> * Thomas Sun (tom dot ssf at gmail dot com) >>>>>>>> * Yaliang Wang (yaliang dot w dot wang at ieee dot org) >>>>>>>> >>>>>>>> ## Affiliations >>>>>>>> >>>>>>>> * Andrew Jorgensen (Google) >>>>>>>> * Ashvin Agrawal (Microsoft) >>>>>>>> * Avrilia Floratou (Microsoft) >>>>>>>> * Bill Graham (Twitter) >>>>>>>> * Brian Hatfield (Google) >>>>>>>> * Chris Kellogg (Twitter) >>>>>>>> * Huijun Wu (Twitter) >>>>>>>> * Karthik Ramasamy (Streamlio) >>>>>>>> * Maosong Fu (Twitter) >>>>>>>> * Neng Lu (Twitter) >>>>>>>> * Runhang Li (Twitter) >>>>>>>> * Sanjeev Kulkarni (Streamlio) >>>>>>>> * Supun Kamburugamuve (Indiana University) >>>>>>>> * Thomas Sun (Twitter) >>>>>>>> * Yaliang Wang (Twitter) >>>>>>>> >>>>>>>> ## Sponsors >>>>>>>> >>>>>>>> ### Champion >>>>>>>> >>>>>>>> * Julien Le Dem (julien at apache dot org) >>>>>>>> >>>>>>>> ### Nominated Mentors >>>>>>>> >>>>>>>> * Jake Farrell (jfarrell at apache dot org) >>>>>>>> * Jacques Nadeau (jacques at apache dot org) >>>>>>>> * Julien Le Dem (julien at apache dot org) >>>>>>>> >>>>>>>> ### Sponsoring Entity >>>>>>>> >>>>>>>> The Apache Incubator >>>>>>>> >>>>>>>> ### Footnotes >>>>>>>> >>>>>>>> 1 - Papers detailing Heron are available at >>>> http://dl.acm.org/citation >>>>>> . >>>>>>>> cfm?id=2742788 and http://sites.computer.org/debull/A15dec/p15.pdf. >>>>>>>> 2 - http://home.apache.org/phonebook.html?uid=billgraham >>>>>>>> 3 - http://home.apache.org/phonebook.html?uid=ashvin >>>>>>>> 4 - http://home.apache.org/phonebook.html?uid=supun >>>>>>> >>>>>>> ------------------------------------------------------------ >> --------- >>>>>>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >>>>>>> For additional commands, e-mail: general-h...@incubator.apache.org >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Supun Kamburugamuve >>>>> Member, Apache Software Foundation; http://www.apache.org >>>>> E-mail: supun@apache.o <supu...@gmail.com>rg; Mobile: +1 812 219 2563 >>>>> <(812)%20219-2563> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >>>> For additional commands, e-mail: general-h...@incubator.apache.org >>>> >>>> >>> >>> >>> -- >>> ~/William >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> For additional commands, e-mail: general-h...@incubator.apache.org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
