+1 (binding) On Thu, Dec 1, 2016 at 8:47 AM, Andrew Purtell <andrew.purt...@gmail.com> wrote:
> +1 (binding) > > > On Dec 1, 2016, at 8:35 AM, Felix Cheung <felixche...@apache.org> wrote: > > > > +1 > > > > On Wed, Nov 30, 2016 at 10:40 PM Henry Saputra <henry.sapu...@gmail.com> > > wrote: > > > >> Hi All, > >> > >> As the champion for Griffin, I would like to start VOTE to bring the > >> project as Apache incubator podling. > >> > >> Here is the direct quote from the abstract: > >> > >> " > >> Griffin is a Data Quality Service platform built on Apache Hadoop and > >> Apache Spark. It provides a framework process for defining data > >> quality model, executing data quality measurement, automating data > >> profiling and validation, as well as a unified data quality > >> visualization across multiple data systems. It tries to address the > >> data quality challenges in big data and streaming context. > >> " > >> > >> Please cast your vote: > >> > >> [ ] +1, bring Griffin into Incubator > >> [ ] +0, I don't care either way, > >> [ ] -1, do not bring Griffin into Incubator, because... > >> > >> This vote will be open at least for 72 hours and only votes from the > >> Incubator PMC are binding. > >> > >> The VOTE will end 12/5 9am PST to pass through weekend. > >> > >> > >> Here is the link to the proposal: > >> > >> https://wiki.apache.org/incubator/GriffinProposal > >> > >> I have copied the proposal below for easy access > >> > >> > >> Thanks, > >> > >> - Henry > >> > >> > >> > >> Griffin Proposal > >> > >> Abstract > >> > >> Griffin is a Data Quality Service platform built on Apache Hadoop and > >> Apache Spark. It provides a framework process for defining data > >> quality model, executing data quality measurement, automating data > >> profiling and validation, as well as a unified data quality > >> visualization across multiple data systems. It tries to address the > >> data quality challenges in big data and streaming context. > >> > >> Proposal > >> > >> Griffin is a open source Data Quality solution for distributed data > >> systems at any scale in both streaming or batch data context. When > >> people use open source products (e.g. Apache Hadoop, Apache Spark, > >> Apache Kafka, Apache Storm), they always need a data quality service > >> to build his/her confidence on data quality processed by those > >> platforms. Griffin creates a unified process to define and construct > >> data quality measurement pipeline across multiple data systems to > >> provide: > >> > >> Automatic quality validation of the data > >> Data profiling and anomaly detection > >> Data quality lineage from upstream to downstream data systems. > >> Data quality health monitoring visualization > >> Shared infrastructure resource management > >> > >> Overview of Griffin > >> > >> Griffin has been deployed in production at eBay serving major data > >> systems, it takes a platform approach to provide generic features to > >> solve common data quality validation pain points. Firstly, user can > >> register the data asset which user wants to do data quality check. The > >> data asset can be batch data in RDBMS (e.g.Teradata), Apache Hadoop > >> system or near real-time streaming data from Apache Kafka, Apache > >> Storm and other real time data platforms. Secondly, user can create > >> data quality model to define the data quality rule and metadata. > >> Thirdly, the model or rule will be executed automatically (by the > >> model engine) to get the sample data quality validation results in a > >> few seconds for streaming data. Finally, user can analyze the data > >> quality results through built-in visualization tool to take actions. > >> > >> Griffin includes: > >> > >> Data Quality Model Engine > >> > >> Griffin is model driven solution, user can choose various data quality > >> dimension to execute his/her data quality validation based on selected > >> target data-set or source data-set ( as the golden reference data). It > >> has a corresponding library supporting it in back-end for the > >> following measurement: > >> > >> Accuracy - Does data reflect the real-world objects or a verifiable > source > >> Completeness - Is all necessary data present > >> Validity - Are all data values within the data domains specified by the > >> business > >> Timeliness - Is the data available at the time needed > >> Anomaly detection - Pre-built algorithm functions for the > >> identification of items, events or observations which do not conform > >> to an expected pattern or other items in a dataset > >> Data Profiling - Apply statistical analysis and assessment of data > >> values within a dataset for consistency, uniqueness and logic. > >> > >> Data Collection Layer > >> > >> We support two kinds of data sources, batch data and real time data. > >> > >> For batch mode, we can collect data source from Apache Hadoop based > >> platform by various data connectors. > >> > >> For real time mode, we can connect with messaging system like Kafka to > >> near real time analysis. > >> > >> Data Process and Storage Layer > >> > >> For batch analysis, our data quality model will compute data quality > >> metrics in our spark cluster based on data source in Apache Hadoop. > >> > >> For near real time analysis, we consume data from messaging system, > >> then our data quality model will compute our real time data quality > >> metrics in our spark cluster. for data storage, we use time series > >> database in our back end to fulfill front end request. > >> > >> Griffin Service > >> > >> We have RESTful web services to accomplish all the functionalities of > >> Griffin, such as register data asset, create data quality model, > >> publish metrics, retrieve metrics, add subscription, etc. So, the > >> developers can develop their own user interface based on these web > >> services. > >> > >> Background > >> > >> At eBay, when people play with big data in Apache Hadoop (or other > >> streaming data), data quality often becomes one big challenge. > >> Different teams have built customized data quality tools to detect and > >> analyze data quality issues within their own domain. We are thinking > >> to take a platform approach to provide shared Infrastructure and > >> generic features to solve common data quality pain points. This would > >> enable us to build trusted data assets. > >> > >> Currently it’s very difficult and costly to do data quality validation > >> when we have big data flow across multi-platforms at eBay (e.g. > >> Oracle, Apache Hadoop, Couchbase, Apache Cassandra, Apache Kafka, > >> MongoDB). Take eBay real time personalization platform as an example. > >> Every day we have to validate data quality status for ~600M records ( > >> imagine we have 150M active users for our website). Data quality often > >> becomes one big challenge both in its streaming and batch pipelines. > >> > >> So we conclude 3 data quality problems at eBay: > >> > >> Lack of end2end unified view of data quality measurement from multiple > >> data sources to target applications, it usually takes a long time to > >> identify and fix poor data quality. > >> How to get data quality measured in streaming mode, we need to have a > >> process and tool to visualize data quality insights through > >> registering dataset which you want to check data quality, creating > >> data quality measurement model, executing the data quality validation > >> job and getting metrics insights for action taking. > >> No Shared platform and API Service, have to apply and manage own > >> hardware and software infrastructure. > >> > >> Rationale > >> > >> The challenge we face at eBay is that our data volume is becoming > >> bigger and bigger, system processes become more complex, while we do > >> not have a unified data quality solution to ensure the trusted data > >> sets which provide confidences on data quality to our data consumers. > >> The key challenges on data quality includes: > >> > >> Existing commercial data quality solution cannot address data quality > >> lineage among systems, cannot scale out to support fast growing data > >> at eBay > >> Existing eBay's domain specific tools take a long time to identify and > >> fix poor data quality when data flowed through multiple systems > >> Business logic becomes complex, requires data quality system much > flexible. > >> > >> Some data quality issues do have business impact on user experiences, > >> revenue, efficiency & compliance. > >> > >> Communication overhead of data quality metrics, typically in a big > >> organization, which involve different teams. > >> > >> The idea of Griffin is to provide Data Quality validation as a > >> Service, to allow data engineers and data consumers to have: > >> > >> Near real-time understanding of the data quality health of your data > >> pipelines with end-to-end monitoring, all in one place. > >> Profiling, detecting and correlating issues and providing > >> recommendations that drive rapid and focused troubleshooting > >> A centralized data quality model management system including rule, > >> metadata, scheduler etc. > >> Native code generation to run everywhere, including Hadoop, Kafka, > Spark, > >> etc. > >> One set of tools to build data quality pipelines across all eBay data > >> platforms. > >> > >> Current Status > >> > >> Meritocracy > >> > >> Griffin has been deployed in production at eBay and provided the > >> centralized data quality service for several eBay systems ( for > >> example, real time personalization platform, eBay real time ID linking > >> platform, Hadoop datasets, Site speed analytics platform). Our aim is > >> to build a diverse developer and user community following the Apache > >> meritocracy model. We will encourage contributions and participation > >> of all types of work, and ensure that contributors are appropriately > >> recognized. > >> > >> Community > >> > >> Currently the project is being developed at eBay. It's only for eBay > >> internal community. Griffin seeks to develop the developer and user > >> communities during incubation. We believe it will grow substantially > >> by becoming an Apache project. > >> > >> Core Developers > >> > >> Griffin is currently being designed and developed by engineers from > >> eBay Inc. – William Guo, Alex Lv, Shawn Sha, Vincent Zhao, John Liu. > >> All of these core developers have deep expertise in Apache Hadoop and > >> the Hadoop Ecosystem in general. > >> > >> Alignment > >> > >> The ASF is a natural host for Griffin given that it is already the > >> home of Hadoop, Beam, HBase, Hive, Storm, Kafka, Spark and other > >> emerging big data products. Those are requiring data quality solution > >> by nature to ensure the data quality which they processed. When people > >> use open source data technology, the big question to them is that how > >> we can ensure the data quality in it. Griffin leverages lot of Apache > >> open-source products. Griffin was designed to enable real time > >> insights into data quality validation by shared Infrastructure and > >> generic features to solve common data quality pain points. > >> > >> Known Risks > >> > >> Orphaned Products > >> > >> The core developers of Griffin team work full time on this project. > >> There is no risk of Griffin getting orphaned since at least one large > >> company (eBay) is extensively using it in their production Hadoop and > >> Spark clusters for multiple data systems. For example, currently there > >> are 4 data systems at eBay (real time personalization platform, eBay > >> real time ID linking platform, Hadoop, Site speed analytics platform) > >> are leveraging Griffin, with more than ~600M records for data quality > >> status validation every day, 35 data sets being monitored, 50+ data > >> quality models have been created. > >> > >> As Griffin is designed to connect many types of data sources, we are > >> very confident that they will use Griffin as a service for ensuring > >> the data quality in open source data ecosystems. We plan to extend and > >> diversify this community further through Apache. > >> > >> Inexperience with Open Source > >> > >> Griffin's core engineers are all active users and followers of open > >> source projects. They are already committers and contributors to the > >> Griffin Github project. All have been involved with the source code > >> that has been released under an open source license, and several of > >> them also have experience developing code in an open source > >> environment. Though the core set of Developers do not have Apache Open > >> Source experience, there are plans to onboard individuals with Apache > >> open source experience on to the project. > >> > >> Homogenous Developers > >> > >> The core developers are from eBay. Apache Incubation process > >> encourages an open and diverse meritocratic community. Griffin intends > >> to make every possible effort to build a diverse, vibrant and involved > >> community. We are committed to recruiting additional committers from > >> other companies based on their contribution to the project. > >> > >> Reliance on Salaried Developers > >> > >> eBay invested in Griffin as a company-wide data quality service > >> platform and some of its key engineers are working full time on the > >> project. they are all paid by eBay. We look forward to other Apache > >> developers and researchers to contribute to the project. > >> > >> Relationships with Other Apache Products > >> > >> Griffin has a strong relationship and dependency with Apache Hadoop, > >> Apache HBase, Apache Spark, Apache Kafka and Apache Storm, Apache > >> Hive. In addition, since there is a growing need for data quality > >> solution for open source platform (e.g. Hadoop, Kafka, Spark etc), > >> being part of Apache’s Incubation community, could help with a closer > >> collaboration among these four projects and as well as others. > >> > >> Documentation > >> > >> Information about Griffin can be found at https://github.com/eBay/ > griffin > >> > >> Initial Source > >> > >> Griffin has been under development since early 2016 by a team of > >> engineers at eBay Inc. It is currently hosted on Github.com under an > >> Apache license 2.0 at https://github.com/eBay/griffin . Once in > >> incubation we will be moving the code base to apache git library. > >> > >> External Dependencies > >> > >> Griffin has the following external dependencies. > >> > >> Basic > >> > >> JDK 1.7+ > >> Scala > >> Apache Maven > >> JUnit > >> Log4j > >> Slf4j > >> Apache Commons > >> > >> Hadoop > >> > >> Apache Hadoop > >> Apache HBase > >> Apache Hive > >> > >> DB > >> > >> InfluxData > >> > >> Apache Spark > >> > >> Spark Core Library > >> > >> REST Service > >> > >> Jersey > >> Spring MVC > >> > >> Web frontend > >> > >> AngularJS > >> jQuery > >> Bootstrap > >> RequireJS > >> eCharts > >> Font Awesome > >> > >> Cryptography > >> > >> Currently there's no cryptography in Griffin. > >> > >> Required Resources > >> > >> Mailing List > >> > >> We currently use eBay mail box to communicate, but we'd like to move > >> that to ASF maintained mailing lists. > >> > >> Current mailing list: ebay-griffin-d...@googlegroups.com > >> > >> Proposed ASF maintained lists: > >> > >> priv...@griffin.incubator.apache.org > >> > >> d...@griffin.incubator.apache.org > >> > >> comm...@griffin.incubator.apache.org > >> > >> Subversion Directory > >> > >> Git is the preferred source control system. > >> > >> Issue Tracking > >> > >> JIRA > >> > >> Other Resources > >> > >> The existing code already has unit tests so we will make use of > >> existing Apache continuous testing infrastructure. The resulting load > >> should not be very large. > >> > >> Initial Committers > >> > >> William Go > >> Alex Lv > >> Vincent Zhao > >> Shawn Sha > >> John Liu > >> Liang Shao > >> > >> Affiliations > >> > >> The initial committers are employees of eBay Inc. > >> > >> Sponsors > >> > >> Champion > >> > >> Henry Saputra (hsapu...@apache.org) > >> > >> Nominated Mentors > >> > >> Kasper Sørensen (kasper...@apache.org) > >> > >> Uma Maheswara Rao Gangumalla (umamah...@apache.org) > >> > >> Luciano Resende (luckbr1...@gmail.com) > >> > >> Sponsoring Entity > >> > >> We are requesting the Incubator to sponsor this project. > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > >> For additional commands, e-mail: general-h...@incubator.apache.org > >> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
