Hi Andrei, thanks for offering, we are happy about any help we can get.
On Mon, Nov 4, 2013 at 10:43 AM, Andrei Savu <as...@apache.org> wrote: > IMO it's a good idea to keep this a separate project - the same way we have > Curator for ZooKeeper. > > If you are looking for one more mentor I'm happy to help. > > -- Andrei Savu > > On Sun, Nov 3, 2013 at 11:21 PM, Roman Shaposhnik <r...@apache.org> wrote: > > > Have you guys considered contributing this code directly to Hadoop? > > > > From what you are describing it sounds like a developer friendly library > > wrapped around YARN (along the lines of kitten: > > https://github.com/jwills/kitten). > > > > > > Thanks, > > Roman. > > > > On Tue, Oct 29, 2013 at 12:53 PM, Andreas Neumann <a...@apache.org> > wrote: > > > I would like to propose Weave, an abstraction over Apache Hadoop® YARN > to > > > reduce the complexity of developing distributed applications, as an > > > Apache Incubator > > > podling. > > > > > > The proposal is included in plain text. I would also like to put this > on > > > the wiki, but I appear to lack privileges to create pages. What do I > need > > > to do to get permission? > > > > > > -Andreas. > > > > > > Abstract > > > ======== > > > > > > Weave is an abstraction over Apache Hadoop® YARN that reduces the > > > complexity of developing distributed applications, allowing developers > to > > > focus more on their business logic. > > > > > > Proposal > > > ======== > > > > > > Weave is a set of libraries that reduces the complexity of developing > > > distributed applications. It exposes the distributed capabilities of > > Apache > > > Hadoop® YARN via a simple and intuitive programming model similar to > Java > > > threads. Weave also has built-in capabilities required by many > > distributed > > > applications, such as real-time application logs and metrics > collection, > > > application lifecycle management, and network service discovery. > > > > > > Background > > > ========== > > > > > > Hadoop YARN is a generic cluster resource manager that supports any > type > > of > > > distributed application. However, YARN’s interfaces are too low level > for > > > rapid application development. It requires a great deal of boilerplate > > code > > > even for a simple application, creating a high ramp up cost that can > turn > > > developers away. > > > > > > Weave is designed to improve this situation with a programming model > that > > > makes running distributed applications as easy as running Java threads. > > > With the abstraction provided by Weave, applications can be executed in > > > process threads during development and unit testing and then be > deployed > > to > > > a YARN cluster without any modifications. > > > > > > Weave also has built-in support for real-time application logs and > > metrics > > > collection, delegation token renewal, application lifecycle management, > > and > > > network service discovery. This greatly reduces the pain that > developers > > > face when developing, debugging, deploying and monitoring distributed > > > applications. > > > > > > Weave is not a replacement for YARN, it’s a framework that operates on > > top > > > of YARN. > > > > > > Rationale > > > ========= > > > > > > Developers who write YARN applications typically find themselves > > > implementing the same (or similar) boilerplate code over and over again > > for > > > every application. It makes sense to distill this common code into a > > > reusable set of libraries that is perpetually maintained and improved > by > > a > > > diverse community of developers. > > > > > > Weave’s simple thread-like programming model will enable many Java > > > programmers to develop distributed applications. We believe that this > > > simplicity will attract developers who would otherwise be discouraged > by > > > complexity, and many new use cases will emerge for the usage of YARN. > > > > > > Incubating Weave as an Apache project makes sense because Weave is a > > > framework built on top of YARN, and Weave uses Apache Zookeeper, HDFS, > > > Kafka, and other Apache software (see the External Dependencies > section). > > > > > > Current Status > > > ============== > > > > > > Weave was initially developed at Continuuity. The Weave codebase is > > > currently hosted in a public repository at github.com, which will seed > > the > > > Apache git repository. > > > > > > Meritocracy > > > ----------- > > > Our intent with this incubator proposal is to start building a diverse > > > developer community around Weave following the Apache meritocracy > model. > > > Since Weave was initially developed in early 2013, we have had fast > > > adoption and contributions within Continuuity. We are looking forward > to > > > new contributors. We wish to build a community based on Apache's > > > meritocracy principles, working with those who contribute significantly > > to > > > the project and welcoming them to be committers both during the > > incubation > > > process and beyond. > > > > > > Community > > > --------- > > > Weave is currently being used internally at Continuuity and is at the > > core > > > of our products. We hope to extend our contributor base significantly > and > > > we will invite all who are interested in simplifying the development of > > > distributed applications to participate. > > > > > > Core Developers > > > --------------- > > > Weave is currently being developed by five engineers at Continuuity: > > > Terence Yim, Andreas Neumann, Gary Helmling, Poorna Chandra and Albert > > > Shau. > > > Terence Yim is an Apache committer for Helix, Andreas is an Apache > > > committer and PMC member for Oozie, and Gary Helmling is an Apache > > > committer and PMC member for HBase. Poorna Chandra and Albert Shau have > > > made many contributions to Weave. > > > > > > Alignment > > > --------- > > > The ASF is the natural choice to host the Weave project as its goal of > > > encouraging community-driven open source projects fits with our vision > > for > > > Weave. > > > > > > Additionally, many other projects with which we are familiar and expect > > > Weave to integrate with, such as ZooKeeper, YARN, HDFS, log4j, and > others > > > mentioned in the External Dependencies section are Apache projects, and > > > Weave will benefit by close proximity to them. > > > > > > Known Risks > > > =========== > > > > > > Orphaned Products > > > ----------------- > > > There is very little risk of Weave being orphaned, as it is a key part > of > > > Continuuity’s products. The core Weave developers plan to continue to > > work > > > on Weave, and Continuuity has funding in place to support their efforts > > > going forward. > > > Many other Big Data companies can benefit from Weave, and we have > already > > > received interest from various entities that would like to use and > > > contribute to Weave. > > > > > > Inexperience with Open Source > > > ----------------------------- > > > Several of the core developers have experience with open source > > > development. Terence Yim, Andreas Neumann and Gary Helmling are > currently > > > Apache committers for Helix, Oozie and HBase respectively > > > Homogeneous Developers > > > > > > The current core developers are all Continuuity employees. However, we > > > intend to establish a developer community that includes independent and > > > corporate contributors. We are encouraging new contributors via our > > mailing > > > lists, public presentations, and personal contacts, and we will > continue > > to > > > do so. Various entities have already expressed interest in becoming > > > involved with Weave. > > > > > > Reliance on Salaried Developers > > > ------------------------------- > > > Currently, these developers are paid to work on Weave. Once the project > > has > > > built a community, we expect to attract committers, developers and > > > community other the current core developers. However, because > Continuuity > > > products use Weave internally, the reliance on salaried developers is > > > unlikely to change, at least in the near term. > > > > > > Relationships with Other Apache Products > > > ---------------------------------------- > > > Weave is deeply integrated with Apache projects. Weave uses Apache YARN > > as > > > its underlying resource management and task scheduling system and > Apache > > > Zookeeper for coordination. In addition, Weave uses Apache HDFS and > > Apache > > > Kafka. A number of other Apache projects are Weave dependencies and are > > > listed in the External Dependencies section. > > > > > > An Excessive Fascination with the Apache Brand > > > ---------------------------------------------- > > > While we respect the reputation of the Apache brand and have no doubt > > that > > > it will attract contributors and users, our interest is primarily to > give > > > Weave a solid home as an open source project following an established > > > development model. We have also given additional reasons in the > Rationale > > > and Alignment sections. > > > > > > Documentation > > > ============= > > > > > > https://github.com/continuuity/weave > > > > > > Initial Source > > > ============== > > > > > > The Weave codebase is currently hosted at > > > https://github.com/continuuity/weave. > > > > > > External Dependencies > > > ===================== > > > > > > The dependencies all have Apache-compatible licenses: > > > • avro (Apache 2.0) > > > • hadoop (Apache 2.0) > > > • gson (Apache 2.0) > > > • guava-libraries (Apache 2.0) > > > • hbase (Apache 2.0) > > > • hdfs (Apache 2.0) > > > • kafka (Apache 2.0) > > > • netty (Apache 2.0) > > > • snappy-java (Apache 2.0) > > > • yarn (Apache 2.0) > > > • zookeeper (Apache 2.0) > > > • asm (BSD) > > > • junit (EPL v1.0) > > > • logback (EPL v1.0 ) > > > • slf4j (MIT) > > > > > > Cryptography > > > ============ > > > > > > Weave will depend on secure Hadoop, which can optionally use Kerberos. > > > > > > Required Resources > > > ================== > > > > > > Mailing Lists > > > ------------- > > > - weave-private for private PMC discussions (with moderated > > subscriptions) > > > - weave-dev for technical discussions among contributors > > > - weave-commits for notification about commits > > > > > > Subversion Directory > > > -------------------- > > > Git is the preferred source control system: git://git.apache.org/weave > > > > > > Issue Tracking > > > -------------- > > > JIRA Weave (WEAVE) > > > > > > Other Resources > > > --------------- > > > The existing code already has unit tests, so we would like a Hudson > > > instance to run them whenever a new patch is submitted. This can be > added > > > after project creation. > > > > > > Initial Committers > > > ================== > > > > > > • Terence Yim > > > • Andreas Neumann > > > • Gary Helmling > > > • Poorna Chandra > > > • Albert Shau > > > > > > Affiliations > > > ============ > > > > > > • Terence Yim (Continuuity) > > > • Andreas Neumann (Continuuity) > > > • Gary Helmling (Continuuity) > > > • Poorna Chandra (Continuuity) > > > • Albert Shau (Continuuity) > > > > > > Sponsors > > > ======== > > > > > > Champion > > > -------- > > > Vinod K <vinodkv at apache dot org> (Apache Member) > > > > > > Nominated Mentors > > > ----------------- > > > • Arun C Murthy <acmurthy at apache dot org> > > > • Tom White <tomwhite at apache dot org> > > > • Patrick Hunt <phunt at apache dot org> > > > > > > Sponsoring Entity > > > ----------------- > > > We are requesting that the Incubator sponsor this project. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > >
