Off course, my + 1 On Thu, Oct 26, 2017 at 12:31 PM, Luciano Resende <luckbr1...@gmail.com> wrote:
> Now that the discussion thread on the Crail proposal has ended, please > vote on accepting Crail into into the Apache Incubator. > > The ASF voting rules are described at: > http://www.apache.org/foundation/voting.html > > A vote for accepting a new Apache Incubator podling is a majority vote > for which only Incubator PMC member votes are binding. > > Votes from other people are also welcome as an indication of peoples > enthusiasm (or lack thereof). > > Please do not use this VOTE thread for discussions. > If needed, start a new thread instead. > > This vote will run for at least 72 hours. Please VOTE as follows > [] +1 Accept Crail into the Apache Incubator > [] +0 Abstain. > [] -1 Do not accept Crail into the Apache Incubator because ... > > The proposal below is also on the wiki: > https://wiki.apache.org/incubator/CrailProposal > > === > > Abstract > > Crail is a storage platform for sharing performance critical data in > distributed data processing jobs at very high speed. Crail is built > entirely upon principles of user-level I/O and specifically targets data > center deployments with fast network and storage hardware (e.g., 100Gbps > RDMA, plenty of DRAM, NVMe flash, etc.) as well as new modes of operation > such resource disaggregation or serverless computing. Crail is written in > Java and integrates seamlessly with the Apache data processing ecosystem. > It can be used as a backbone to accelerate high-level data operations such > as shuffle or broadcast, or as a cache to store hot data that is queried > repeatedly, or as a storage platform for sharing inter-job data in complex > multi-job pipelines, etc. > > Proposal > > Crail enables Apache data processing frameworks to run efficiently in next > generation data centers using fast storage and network hardware in > combination with resource (e.g., DRAM, Flash) disaggregation. > > Background > > Crail started as a research project at the IBM Zurich Research Laboratory > around 2014 aiming to integrate high-speed I/O hardware effectively into > large scale data processing systems. > > Rational > > During the last decade, I/O hardware has undergone rapid performance > improvements, typically in the order of magnitudes. Modern day networking > and storage hardware can deliver 100+ Gbps (10+ GBps) bandwidth with a few > microseconds of access latencies. However, despite such progress in raw I/O > performance, effectively leveraging modern hardware in data processing > frameworks remains challenging. In most of the cases, upgrading to high-end > networking or storage hardware has very little effect on the performance of > analytics workloads. The problem comes from heavily layered software > imposing overheads such as deep call stacks, unnecessary data copies, > thread contention, etc. These problems have already been addressed at the > operating system level with new I/O APIs such as RDMA verbs, NVMe, etc., > allowing applications to bypass software layers during I/O operations. > Distributed data processing frameworks on the other hand, are typically > implemented on legacy I/O interfaces such as such as sockets or block > storage. These interfaces have been shown to be insufficient to deliver the > full hardware performance. Yet, to the best of our knowledge, there are no > active and systematic efforts to integrate these new user level I/O APIs > into Apache software frameworks. This problem affects all end-users and > organizations that use Apache software. We expect them to see > unsatisfactory small performance gains when upgrading their networking and > storage hardware. > > Crail solves this problem by providing an efficient storage platform built > upon user-level I/O, thus, bypassing layers such as JVM and OS during I/O > operations. Moreover, Crail directly leverages the specific hardware > features of RDMA and NVMe to provide a better integration with high-level > data operations in Apache compute frameworks. As a consequence, Crail > enables users to run larger, more complex queries against ever increasing > amounts of data at a speed largely determined by the deployed hardware. > Crail is generic solution that integrates well with the Apache ecosystem > including frameworks like Spark, Hadoop, Hive, etc. > > Initial Goals > > The initial goals to move Crail to the Apache Incubator is to broaden the > community, and foster contributions from developers to leverage Crail in > various data processing frameworks and workloads. Ultimately, the goal for > Crail is to become the de-facto standard platform for storing temporary > performance critical data in distributed data processing systems. > > Current Status > > The initial code has been developed at the IBM Zurich Research Center and > has recently been made available in GitHub under the Apache Software > License 2.0. The Project currently has explicit support for Spark and > Hadoop. Project documentation is available on the website www.crail.io. > There is also a public forum for discussions related to Crail available at > https://groups.google.com/forum/#!forum/zrlio-users. > > Mericrotacy > > The current developers are familiar with the meritocratic open source > development process at Apache. Over the last year, the project has gathered > interest at GitHub and several companies have already expressed interest in > the project. We plan to invest in supporting a meritocracy by inviting > additional developers to participate. > > Community > > The need for a generic solution to integrate high-performance I/O hardware > in the open source is tremendous, so there is a potential for a very large > community. We believe that Crail’s extensible architecture and its > alignment with the Apache Ecosystem will further encourage community > participation. We expect that over time Crail will attract a large > community. > > Alignment > > Crail is written in Java and is built for the Apache data processing > ecosystem. The basic storage services of Crail can be used seamlessly from > Spark, Hadoop, Storm. The enhanced storage services require dedicated data > processing specific binding, which currently are available only for Spark. > We think that moving Crail to the Apache incubator will help to extend > Crail’s support for different data processing frameworks. > > Known Risks > > To-date, development has been sponsored by IBM and coordinated mostly by > the core team of researchers at the IBM Zurich Research Center. For Crail > to fully transition to an "Apache Way" governance model, it needs to start > embracing the meritocracy-centric way of growing the community of > contributors. > > Orphaned Products > > The Crail developers have a long-term interest in use and maintenance of > the code and there is also hope that growing a diverse community around the > project will become a guarantee against the project becoming orphaned. We > feel that it is also important to put formal governance in place both for > the project and the contributors as the project expands. We feel ASF is the > best location for this. > > Inexperience with Open Source > > Several of the initial committers are experienced open source developers > (Linux Kernel, DPDK, etc.). > > Relationships with Other Apache Products > > As of now, Crail has been tested with Spark, Hadoop and Hive, but it is > designed to integrate with any of the Apache data processing frameworks. > > Homogeneous Developers > > The project already has a diverse developer base including contributions > from organizations and public developers. > > An Excessive Fascination with the Apache Brand > > Crail solves a real need for a generic approach to leverage modern network > and storage hardware effectively in the Apache Hadoop and Spark ecosystems. > Our rationale for developing Crail as an Apache project is detailed in the > Rationale section. We believe that the Apache brand and community process > will help to us to engage a larger community and facilitate closer ties > with various Apache data processing projects. > > Documentation > > Documentation regarding Crail is available at www.crail.io > > Initial Source > > Initial source is available on GitHub under the Apache License 2.0: > > https://github.com/zrlio/crail > External Dependencies > > Crail is written in Java and currently supports Apache Hadoop MapReduce > and Apache Spark runtimes. To the best of our knowledge, all dependencies > of Crail are distributed under Apache compatible licenses. > > Required Resource > > Mailing lists > > priv...@crail.incubator.apache.org > d...@crail.incubator.apache.org > comm...@crail.incubator.apache.org > Git repository > > https://git-wip-us.apache.org/repos/asf/incubator-crail.git > Issue Tracking > > JIRA (Crail) > Initial Committers > > Patrick Stuedi <stu AT ibm DOT zurich DOT com> > Animesh Trivedi <atr AT ibm DOT zurich DOT com> > Jonas Pfefferle <jpf AT ibm DOT zurich DOT com> > Bernard Metzler <bmt AT ibm DOT zurich DOT com> > Michael Kaufmann <kau AT ibm DOT zurich DOT com> > Adrian Schuepbach <dri AT ibm DOT zurich DOT com> > Patrick McArthur <patrick AT patrickmcarthur DOT net> > Ana Klimovic <anakli AT stanford DOT edu> > Yuval Degani <yuvaldeg AT mellanox DOT com> > Vu Pham <vuhuong AT mellanox DOT com> > Affiliations > > IBM (Patrick, Stuedi, Animesh Trivedi, Jonas Pfefferle, Bernard Metzler, > Michael Kaufmann, Adrian Schuepbach) > University of New Hampshire (Patrick McArthur) > Stanford University (Ana Klimovic) > Mellanox (Yuval Degani, Vu Pham) > Sponsors > > Champion > > Luciano Resende <lresende AT apache DOT org> > > Nominated Mentors > > Luciano Resende <lresende AT apache DOT org> > > Raphael Bircher <rbircher AT apache DOT org> > > Julian Hyde <jhyde AT apache DOT org> > > Sponsoring Entity > > We would like to propose the Apache Incubator to sponsor this project. > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
