+1 On Mon, Aug 31, 2015 at 7:01 PM, Luke Han <luke...@gmail.com> wrote: > +1 (non-binding) > > > Best Regards! > --------------------- > > Luke Han > > On Tue, Sep 1, 2015 at 9:41 AM, Chris Douglas <cdoug...@apache.org> wrote: > >> +1 -C >> >> On Mon, Aug 31, 2015 at 11:47 AM, Roman Shaposhnik <r...@apache.org> wrote: >> > Following the discussion earlier: >> > http://s.apache.org/Gaf >> > >> > I would like to call a VOTE for accepting HAWQ >> > as a new incubator project. >> > >> > The proposal is available at: >> > https://wiki.apache.org/incubator/HAWQProposal >> > and is also included at the bottom of this email. >> > >> > Vote is open until at least Thu, 3 September 2015, 23:59:00 PST >> > >> > [ ] +1 accept HAWQ into the Apache Incubator >> > [ ] ±0 >> > [ ] -1 because... >> > >> > Thanks, >> > Roman. >> > >> > == Abstract == >> > >> > HAWQ is an advanced enterprise SQL on Hadoop analytic engine built >> > around a robust and high-performance massively-parallel processing >> > (MPP) SQL framework evolved from Pivotal Greenplum DatabaseⓇ. >> > >> > HAWQ runs natively on Apache HadoopⓇ clusters by tightly integrating >> > with HDFS and YARN. HAWQ supports multiple Hadoop file formats such as >> > Apache Parquet, native HDFS, and Apache Avro. HAWQ is configured and >> > managed as a Hadoop service in Apache Ambari. HAWQ is 100% ANSI SQL >> > compliant (supporting ANSI SQL-92, SQL-99, and SQL-2003, plus OLAP >> > extensions) and supports open database connectivity (ODBC) and Java >> > database connectivity (JDBC), as well. Most business intelligence, >> > data analysis and data visualization tools work with HAWQ out of the >> > box without the need for specialized drivers. >> > >> > A unique aspect of HAWQ is its integration of statistical and machine >> > learning capabilities that can be natively invoked from SQL or (in the >> > context of PL/Python, PL/Java or PL/R) in massively parallel modes and >> > applied to large data sets across a Hadoop cluster. These capabilities >> > are provided through MADlib – an existing open source, parallel >> > machine-learning library. Given the close ties between the two >> > development communities, the MADlib community has expressed interest >> > in joining HAWQ on its journey into the ASF Incubator and will be >> > submitting a separate, concurrent proposal. >> > >> > HAWQ will provide more robust and higher performing options for Hadoop >> > environments that demand best-in-class data analytics for business >> > critical purposes. HAWQ is implemented in C and C++. >> > >> > HAWQ has a few runtime dependencies licensed under the Cat X list: >> > * gperf (GPL Version 3) >> > * libgsasl (LGPL Version 2.1) >> > * libuuid-2.26 (LGPL Version 2) >> > However, given the runtime (dynamic linking) nature of these >> > dependencies it doesn't represent a problem for HAWQ to be considered >> > an ASF project. >> > >> > == Proposal == >> > The goal of this proposal is to bring the core of Pivotal Software, >> > Inc.’s (Pivotal) Pivotal HAWQⓇ codebase into the Apache Software >> > Foundation (ASF) in order to build a vibrant, diverse and >> > self-governed open source community around the technology. Pivotal has >> > agreed to transfer the brand name "HAWQ" to Apache Software Foundation >> > and will stop using HAWQ to refer to this software if the project gets >> > accepted into the ASF Incubator under the name of "Apache HAWQ >> > (incubating)". Pivotal will continue to market and sell an analytic >> > engine product that includes Apache HAWQ (incubating). While HAWQ is >> > our primary choice for a name of the project, in anticipation of any >> > potential issues with PODLINGNAMESEARCH we have come up with two >> > alternative names: (1) Hornet; or (2) Grove. >> > >> > Pivotal is submitting this proposal to donate the HAWQ source code and >> > associated artifacts (documentation, web site content, wiki, etc.) to >> > the Apache Software Foundation Incubator under the Apache License, >> > Version 2.0 and is asking Incubator PMC to establish an open source >> > community. >> > >> > == Background == >> > While the ecosystem of open source SQL-on-Hadoop solutions is fairly >> > developed by now, HAWQ has several unique features that will set it >> > apart from existing ASF and non-ASF projects. HAWQ made its debut in >> > 2013 as a closed source product leveraging a decade's worth of product >> > development effort invested in Greenplum DatabaseⓇ. Since then HAWQ >> > has rapidly gained a solid customer base and became available on >> > non-Pivotal distributions of Hadoop. >> > In 2015 HAWQ still leverages the rock solid foundation of Greenplum >> > Database, while at the same time embracing elasticity and resource >> > management native to Hadoop applications. This allows HAWQ to provide >> > superior SQL on Hadoop performance, scalability and coverage while >> > also providing massively-parallel machine learning capabilities and >> > support for native Hadoop file formats. In addition, HAWQ's advanced >> > features include support for complex joins, rich and compliant SQL >> > dialect and industry-differentiating data federation capabilities. >> > Dynamic pipelining and pluggable query optimizer architecture enable >> > HAWQ to perform queries on Hadoop with the speed and scalability >> > required for enterprise data warehouse (EDW) workloads. HAWQ provides >> > strong support for low-latency analytic SQL queries, coupled with >> > massively parallel machine learning capabilities. This enables >> > discovery-based analysis of large data sets and rapid, iterative >> > development of data analytics applications that apply deep machine >> > learning – significantly shortening data-driven innovation cycles for >> > the enterprise. >> > >> > Hundreds of companies and thousands of servers are running >> > mission-critical applications today on HAWQ managing over PBs of data. >> > >> > == Rationale == >> > Hadoop and HDFS-based data management architectures continue their >> > expansion into the enterprise. As the amount of data stored on Hadoop >> > clusters grows, unlocking the analytics capabilities and democratizing >> > access to that treasure trove of data becomes one of the key concerns. >> > While Hadoop has no shortage of purposefully designed analytical >> > frameworks, the easiest and most cost-effective way to onboard the >> > largest amount of data consumers is provided by offering SQL APIs for >> > data retrieval at scale. Of course, given the high velocity of >> > innovation happening in the underlying Hadoop ecosystem, any >> > SQL-on-Hadoop solution has to keep up with the community. We strongly >> > believe that in the Big Data space, this can be optimally achieved >> > through a vibrant, diverse, self-governed community collectively >> > innovating around a single codebase while at the same time >> > cross-pollinating with various other data management communities. >> > Apache Software Foundation is the ideal place to meet those ambitious >> > goals. We also believe that our initial experience of bringing Pivotal >> > GemfireⓇ into ASF as Apache Geode (incubating) could be leveraged thus >> > improving the chances of HAWQ becoming a vibrant Apache community. >> > >> > == Initial Goals == >> > Our initial goals are to bring HAWQ into the ASF, transition internal >> > engineering processes into the open, and foster a collaborative >> > development model according to the "Apache Way." Pivotal and its >> > partners plan to develop new functionality in an open, >> > community-driven way. To get there, the existing internal build, test >> > and release processes will be refactored to support open development. >> > >> > == Current Status == >> > Currently, the project code base is commercially licensed and is not >> > available to the general public. The documentation and wiki pages are >> > available at FIXME. Although Pivotal HAWQ was developed as a >> > proprietary, closed-source product, its roots are in the PostgreSQL >> > community and the internal engineering practices adopted by the >> > development team lend themselves well to an open, collaborative and >> > meritocratic environment. >> > >> > The Pivotal HAWQ team has always focused on building a robust end user >> > community of paying and non-paying customers. The existing >> > documentation along with StackOverflow and other similar forums are >> > expected to facilitate conversions between our existing users so as to >> > transform them into an active community of HAWQ members, stakeholders >> > and developers. >> > >> > === Meritocracy === >> > Our proposed list of initial committers include the current HAWQ R&D >> > team, Pivotal Field Engineers, and several existing partners. This >> > group will form a base for the broader community we will invite to >> > collaborate on the codebase. We intend to radically expand the initial >> > developer and user community by running the project in accordance with >> > the "Apache Way". Users and new contributors will be treated with >> > respect and welcomed. By participating in the community and providing >> > quality patches/support that move the project forward, contributors >> > will earn merit. They also will be encouraged to provide non-code >> > contributions (documentation, events, community management, etc.) and >> > will gain merit for doing so. Those with a proven support and quality >> > track record will be encouraged to become committers. >> > >> > === Community === >> > If HAWQ is accepted for incubation, the primary initial goal will be >> > transitioning the core community towards embracing the Apache Way of >> > project governance. We would solicit major existing contributors to >> > become committers on the project from the start. >> > >> > === Core Developers === >> > >> > A few of HAWQ's core developers are skilled in working as part of >> > openly governed Apache communities (mainly around Hadoop ecosystem). >> > That said, most of the core developers are currently NOT affiliated >> > with the ASF and would require new ICLAs before committing to the >> > project. >> > >> > === Alignment === >> > The following existing ASF projects can be considered when reviewing >> > HAWQ proposal: >> > >> > Apache Hadoop is a distributed storage and processing framework for >> > very large datasets, focusing primarily on batch processing for >> > analytic purposes. HAWQ builds on top of two key pieces of Hadoop: >> > YARN and HDFS. HAWQ's community roadmap includes plans for >> > contributing Hadoop around HDFS features and increasing support for C >> > and C++ clients. >> > >> > Apache Spark™ is a fast engine for processing large datasets, >> > typically from a Hadoop cluster, and performing batch, streaming, >> > interactive, or machine learning workloads. Recently, Apache Spark >> > has embraced SQL-like APIs around DataFrames at its core. Because of >> > that we would expect a level of collaboration between the two projects >> > when it comes to query optimization and exposing HAWQ tables to Spark >> > analytical pipelines. >> > >> > Apache Hive™ is a data warehouse software that facilitates querying >> > and managing large datasets residing in distributed storage. Hive >> > provides a mechanism to project structure onto this data and query the >> > data using a SQL-like language called HiveQL. Hive is also providing >> > HCatalog capabilities as table and storage management layer for >> > Hadoop, enabling users with different data processing tools to more >> > easily define structure for the data on the grid. Currently the core >> > Hive and HAWQ are viewed as complimentary solutions, but we expect >> > close integration with HCatalog given its dominant position for >> > metadata management on the Hadoop clusters. >> > >> > Apache Phoenix is a high performance relational database layer over >> > HBase for low latency applications. Given Phoenix's exclusive focus on >> > HBase for its data management backend and its overall architecture >> > around HBase's co-processors, it is unlikely that there will be much >> > collaboration between the two projects. >> > >> > == Known Risks == >> > Development has been sponsored mostly by a single company (or its >> > predecessors) thus far and coordinated mainly by the core Pivotal HAWQ >> > team. >> > >> > For the project to fully transition to the Apache Way governance >> > model, development must shift towards the meritocracy-centric model of >> > growing a community of contributors balanced with the needs for >> > extreme stability and core implementation coherency. >> > >> > The tools and development practices in place for the Pivotal HAWQ >> > product are compatible with the ASF infrastructure and thus we do not >> > anticipate any on-boarding pains. >> > >> > The project currently includes a modified version of PostgreSQL 8.3 >> > source code. Given the ASF's position that the PostgreSQL License is >> > compatible with the Apache License version 2.0, we do NOT anticipate >> > any issues with licensing the code base. However, any new capabilities >> > developed by the HAWQ team once part of the ASF would need to be >> > consumed by the PostgreSQL community under the Apache License version >> > 2.0. >> > >> > === Orphaned products === >> > Pivotal is fully committed to maintaining its position as one of the >> > leading providers of SQL-on-Hadoop solutions and the corresponding >> > Pivotal commercial product will continue to be based on the HAWQ >> > project. Moreover, Pivotal has a vested interest in making HAWQ >> > successful by driving its close integration with both existing >> > projects contributed by Pivotal including Apache Geode (incubating) >> > and MADlib (which is requesting Incubation), and sister ASF projects. >> > We expect this to further reduces the risk of orphaning the product. >> > >> > === Inexperience with Open Source === >> > Pivotal has embraced open source software since its formation by >> > employing contributors/committers and by shepherding open source >> > projects like Cloud Foundry, Spring, RabbitMQ and MADlib. Individuals >> > working at Pivotal have experience with the formation of vibrant >> > communities around open technologies with the Cloud Foundry >> > Foundation, and continuing with the creation of a community around >> > Apache Geode (incubating). Although some of the initial committers >> > have not had the experience of developing entirely open source, >> > community-driven projects, we expect to bring to bear the open >> > development practices that have proven successful on longstanding >> > Pivotal open source projects to the HAWQ community. Additionally, >> > several ASF veterans have agreed to mentor the project and are listed >> > in this proposal. The project will rely on their collective guidance >> > and wisdom to quickly transition the entire team of initial committers >> > towards practicing the Apache Way. >> > >> > === Homogeneous Developers === >> > While most of the initial committers are employed by Pivotal, we have >> > already seen a healthy level of interest from existing customers and >> > partners. We intend to convert that interest directly into >> > participation and will be investing in activities to recruit >> > additional committers from other companies. >> > >> > === Reliance on Salaried Developers === >> > Most of the contributors are paid to work in the Big Data space. While >> > they might wander from their current employers, they are unlikely to >> > venture far from their core expertise and thus will continue to be >> > engaged with the project regardless of their current employers. >> > >> > === Relationships with Other Apache Products === >> > As mentioned in the Alignment section, HAWQ may consider various >> > degrees of integration and code exchange with Apache Hadoop, Apache >> > Spark and Apache Hive projects. We expect integration points to be >> > inside and outside the project. We look forward to collaborating with >> > these communities as well as other communities under the Apache >> > umbrella. >> > >> > === An Excessive Fascination with the Apache Brand === >> > While we intend to leverage the Apache ‘branding’ when talking to >> > other projects as testament of our project’s ‘neutrality’, we have no >> > plans for making use of Apache brand in press releases nor posting >> > billboards advertising acceptance of HAWQ into Apache Incubator. >> > >> > == Documentation == >> > The documentation is currently available at http://hawq.docs.pivotal.io/ >> > >> > == Initial Source == >> > Initial source code will be available immediately after Incubator PMC >> > approves HAWQ joining the Incubator and will be licensed under the >> > Apache License v2. >> > >> > == Source and Intellectual Property Submission Plan == >> > As soon as HAWQ is approved to join the Incubator, the source code >> > will be transitioned via an exhibit to Pivotal's current Software >> > Grant Agreement onto ASF infrastructure and in turn made available >> > under the Apache License, version 2.0. We know of no legal >> > encumberments that would inhibit the transfer of source code to the >> > ASF. >> > >> > == External Dependencies == >> > >> > Runtime dependencies: >> > * gimli (BSD) >> > * openldap (The OpenLDAP Public License) >> > * openssl (OpenSSL License and the Original SSLeay License, BSD style) >> > * proj (MIT) >> > * yaml (Creative Commons Attribution 2.0 License) >> > * python (Python Software Foundation License Version 2) >> > * apr-util (Apache Version 2.0) >> > * bzip2 (BSD-style License) >> > * curl (MIT/X Derivate License) >> > * gperf (GPL Version 3) >> > * protobuf (Google) >> > * libevent (BSD) >> > * json-c (https://github.com/json-c/json-c/blob/master/COPYING) >> > * krb5 (MIT) >> > * pcre (BSD) >> > * libedit (BSD) >> > * libxml2 (MIT) >> > * zlib (Permissive Free Software License) >> > * libgsasl (LGPL Version 2.1) >> > * thrift (Apache Version 2.0) >> > * snappy (Apache Version 2.0 (up to 1.0.1)/New BSD) >> > * libuuid-2.26 (LGPL Version 2) >> > * apache hadoop (Apache Version 2.0) >> > * apache avro (Apache Version 2.0) >> > * glog (BSD) >> > * googlemock (BSD) >> > >> > Build only dependencies: >> > * ant (Apache Version 2.0) >> > * maven (Apache Version 2.0) >> > * cmake (BSD) >> > >> > Test only dependencies: >> > * googletest (BSD) >> > >> > Cryptography N/A >> > >> > == Required Resources == >> > >> > === Mailing lists === >> > * priv...@hawq.incubator.apache.org (moderated subscriptions) >> > * comm...@hawq.incubator.apache.org >> > * d...@hawq.incubator.apache.org >> > * iss...@hawq.incubator.apache.org >> > * u...@hawq.incubator.apache.org >> > >> > === Git Repository === >> > https://git-wip-us.apache.org/repos/asf/incubator-hawq.git >> > >> > === Issue Tracking === >> > JIRA Project HAWQ (HAWQ) >> > >> > === Other Resources === >> > >> > Means of setting up regular builds for HAWQ on builds.apache.org will >> > require integration with Docker support. >> > >> > == Initial Committers == >> > * Lirong Jian >> > * Hubert Huan Zhang >> > * Radar Da Lei >> > * Ivan Yanqing Weng >> > * Zhanwei Wang >> > * Yi Jin >> > * Lili Ma >> > * Jiali Yao >> > * Zhenglin Tao >> > * Ruilong Huo >> > * Ming Li >> > * Wen Lin >> > * Lei Chang >> > * Alexander V Denissov >> > * Newton Alex >> > * Oleksandr Diachenko >> > * Jun Aoki >> > * Bhuvnesh Chaudhary >> > * Vineet Goel >> > * Shivram Mani >> > * Noa Horn >> > * Sujeet S Varakhedi >> > * Junwei (Jimmy) Da >> > * Ting (Goden) Yao >> > * Mohammad F (Foyzur) Rahman >> > * Entong Shen >> > * George C Caragea >> > * Amr El-Helw >> > * Mohamed F Soliman >> > * Venkatesh (Venky) Raghavan >> > * Carlos Garcia >> > * Zixi (Jesse) Zhang >> > * Michael P Schubert >> > * C.J. Jameson >> > * Jacob Frank >> > * Ben Calegari >> > * Shoabe Shariff >> > * Rob Day-Reynolds >> > * Mel S Kiyama >> > * Charles Alan Litzell >> > * David Yozie >> > * Ed Espino >> > * Caleb Welton >> > * Parham Parvizi >> > * Dan Baskette >> > * Christian Tzolov >> > * Tushar Pednekar >> > * Greg Chase >> > * Chloe Jackson >> > * Michael Nixon >> > * Roman Shaposhnik >> > * Alan Gates >> > * Owen O'Malley >> > * Thejas Nair >> > * Don Bosco Durai >> > * Konstantin Boudnik >> > * Sergey Soldatov >> > * Atri Sharma >> > >> > == Affiliations == >> > * Barclays: Atri Sharma >> > * Bloomberg: Justin Erenkrantz >> > * Hortonworks: Alan Gates, Owen O'Malley, Thejas Nair, Don Bosco Durai >> > * WANDisco: Konstantin Boudnik, Sergey Soldatov >> > * Pivotal: everyone else on this proposal >> > >> > == Sponsors == >> > >> > === Champion === >> > Roman Shaposhnik >> > >> > === Nominated Mentors === >> > >> > The initial mentors are listed below: >> > * Alan Gates - Apache Member, Hortonworks >> > * Owen O'Malley - Apache Member, Hortonworks >> > * Thejas Nair - Apache Member, Hortonworks >> > * Konstantin Boudnik - Apache Member, WANDisco >> > * Roman Shaposhnik - Apache Member, Pivotal >> > * Justin Erenkrantz - Apache Member, Bloomberg >> > >> > === Sponsoring Entity === >> > We would like to propose Apache incubator to sponsor this project. >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> > For additional commands, e-mail: general-h...@incubator.apache.org >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> For additional commands, e-mail: general-h...@incubator.apache.org >> >>
--------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
