Have you guys considered contributing this code directly to Hadoop? >From what you are describing it sounds like a developer friendly library wrapped around YARN (along the lines of kitten: https://github.com/jwills/kitten).
Thanks, Roman. On Tue, Oct 29, 2013 at 12:53 PM, Andreas Neumann <a...@apache.org> wrote: > I would like to propose Weave, an abstraction over Apache Hadoop® YARN to > reduce the complexity of developing distributed applications, as an > Apache Incubator > podling. > > The proposal is included in plain text. I would also like to put this on > the wiki, but I appear to lack privileges to create pages. What do I need > to do to get permission? > > -Andreas. > > Abstract > ======== > > Weave is an abstraction over Apache Hadoop® YARN that reduces the > complexity of developing distributed applications, allowing developers to > focus more on their business logic. > > Proposal > ======== > > Weave is a set of libraries that reduces the complexity of developing > distributed applications. It exposes the distributed capabilities of Apache > Hadoop® YARN via a simple and intuitive programming model similar to Java > threads. Weave also has built-in capabilities required by many distributed > applications, such as real-time application logs and metrics collection, > application lifecycle management, and network service discovery. > > Background > ========== > > Hadoop YARN is a generic cluster resource manager that supports any type of > distributed application. However, YARN’s interfaces are too low level for > rapid application development. It requires a great deal of boilerplate code > even for a simple application, creating a high ramp up cost that can turn > developers away. > > Weave is designed to improve this situation with a programming model that > makes running distributed applications as easy as running Java threads. > With the abstraction provided by Weave, applications can be executed in > process threads during development and unit testing and then be deployed to > a YARN cluster without any modifications. > > Weave also has built-in support for real-time application logs and metrics > collection, delegation token renewal, application lifecycle management, and > network service discovery. This greatly reduces the pain that developers > face when developing, debugging, deploying and monitoring distributed > applications. > > Weave is not a replacement for YARN, it’s a framework that operates on top > of YARN. > > Rationale > ========= > > Developers who write YARN applications typically find themselves > implementing the same (or similar) boilerplate code over and over again for > every application. It makes sense to distill this common code into a > reusable set of libraries that is perpetually maintained and improved by a > diverse community of developers. > > Weave’s simple thread-like programming model will enable many Java > programmers to develop distributed applications. We believe that this > simplicity will attract developers who would otherwise be discouraged by > complexity, and many new use cases will emerge for the usage of YARN. > > Incubating Weave as an Apache project makes sense because Weave is a > framework built on top of YARN, and Weave uses Apache Zookeeper, HDFS, > Kafka, and other Apache software (see the External Dependencies section). > > Current Status > ============== > > Weave was initially developed at Continuuity. The Weave codebase is > currently hosted in a public repository at github.com, which will seed the > Apache git repository. > > Meritocracy > ----------- > Our intent with this incubator proposal is to start building a diverse > developer community around Weave following the Apache meritocracy model. > Since Weave was initially developed in early 2013, we have had fast > adoption and contributions within Continuuity. We are looking forward to > new contributors. We wish to build a community based on Apache's > meritocracy principles, working with those who contribute significantly to > the project and welcoming them to be committers both during the incubation > process and beyond. > > Community > --------- > Weave is currently being used internally at Continuuity and is at the core > of our products. We hope to extend our contributor base significantly and > we will invite all who are interested in simplifying the development of > distributed applications to participate. > > Core Developers > --------------- > Weave is currently being developed by five engineers at Continuuity: > Terence Yim, Andreas Neumann, Gary Helmling, Poorna Chandra and Albert > Shau. > Terence Yim is an Apache committer for Helix, Andreas is an Apache > committer and PMC member for Oozie, and Gary Helmling is an Apache > committer and PMC member for HBase. Poorna Chandra and Albert Shau have > made many contributions to Weave. > > Alignment > --------- > The ASF is the natural choice to host the Weave project as its goal of > encouraging community-driven open source projects fits with our vision for > Weave. > > Additionally, many other projects with which we are familiar and expect > Weave to integrate with, such as ZooKeeper, YARN, HDFS, log4j, and others > mentioned in the External Dependencies section are Apache projects, and > Weave will benefit by close proximity to them. > > Known Risks > =========== > > Orphaned Products > ----------------- > There is very little risk of Weave being orphaned, as it is a key part of > Continuuity’s products. The core Weave developers plan to continue to work > on Weave, and Continuuity has funding in place to support their efforts > going forward. > Many other Big Data companies can benefit from Weave, and we have already > received interest from various entities that would like to use and > contribute to Weave. > > Inexperience with Open Source > ----------------------------- > Several of the core developers have experience with open source > development. Terence Yim, Andreas Neumann and Gary Helmling are currently > Apache committers for Helix, Oozie and HBase respectively > Homogeneous Developers > > The current core developers are all Continuuity employees. However, we > intend to establish a developer community that includes independent and > corporate contributors. We are encouraging new contributors via our mailing > lists, public presentations, and personal contacts, and we will continue to > do so. Various entities have already expressed interest in becoming > involved with Weave. > > Reliance on Salaried Developers > ------------------------------- > Currently, these developers are paid to work on Weave. Once the project has > built a community, we expect to attract committers, developers and > community other the current core developers. However, because Continuuity > products use Weave internally, the reliance on salaried developers is > unlikely to change, at least in the near term. > > Relationships with Other Apache Products > ---------------------------------------- > Weave is deeply integrated with Apache projects. Weave uses Apache YARN as > its underlying resource management and task scheduling system and Apache > Zookeeper for coordination. In addition, Weave uses Apache HDFS and Apache > Kafka. A number of other Apache projects are Weave dependencies and are > listed in the External Dependencies section. > > An Excessive Fascination with the Apache Brand > ---------------------------------------------- > While we respect the reputation of the Apache brand and have no doubt that > it will attract contributors and users, our interest is primarily to give > Weave a solid home as an open source project following an established > development model. We have also given additional reasons in the Rationale > and Alignment sections. > > Documentation > ============= > > https://github.com/continuuity/weave > > Initial Source > ============== > > The Weave codebase is currently hosted at > https://github.com/continuuity/weave. > > External Dependencies > ===================== > > The dependencies all have Apache-compatible licenses: > • avro (Apache 2.0) > • hadoop (Apache 2.0) > • gson (Apache 2.0) > • guava-libraries (Apache 2.0) > • hbase (Apache 2.0) > • hdfs (Apache 2.0) > • kafka (Apache 2.0) > • netty (Apache 2.0) > • snappy-java (Apache 2.0) > • yarn (Apache 2.0) > • zookeeper (Apache 2.0) > • asm (BSD) > • junit (EPL v1.0) > • logback (EPL v1.0 ) > • slf4j (MIT) > > Cryptography > ============ > > Weave will depend on secure Hadoop, which can optionally use Kerberos. > > Required Resources > ================== > > Mailing Lists > ------------- > - weave-private for private PMC discussions (with moderated subscriptions) > - weave-dev for technical discussions among contributors > - weave-commits for notification about commits > > Subversion Directory > -------------------- > Git is the preferred source control system: git://git.apache.org/weave > > Issue Tracking > -------------- > JIRA Weave (WEAVE) > > Other Resources > --------------- > The existing code already has unit tests, so we would like a Hudson > instance to run them whenever a new patch is submitted. This can be added > after project creation. > > Initial Committers > ================== > > • Terence Yim > • Andreas Neumann > • Gary Helmling > • Poorna Chandra > • Albert Shau > > Affiliations > ============ > > • Terence Yim (Continuuity) > • Andreas Neumann (Continuuity) > • Gary Helmling (Continuuity) > • Poorna Chandra (Continuuity) > • Albert Shau (Continuuity) > > Sponsors > ======== > > Champion > -------- > Vinod K <vinodkv at apache dot org> (Apache Member) > > Nominated Mentors > ----------------- > • Arun C Murthy <acmurthy at apache dot org> > • Tom White <tomwhite at apache dot org> > • Patrick Hunt <phunt at apache dot org> > > Sponsoring Entity > ----------------- > We are requesting that the Incubator sponsor this project. --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
