Hi Yaniv Thanks, i would find some time to try it.
Is there any deployment UI from Amaterasu for easier using? Regards Liang Yaniv Rodenski wrote > Hi Liang, > > We are currently working on YARN support, but it's not quite there yet. > > What you can do at this point (as a workaround) is setup a single node > Mesos cluster and configure it to use your YARN cluster. > > If you would like to try that LMK, and I'll help you set it up (it's not > that straight forward). > > Cheers, > Yaniv > > On Wed, 8 Mar 2017 at 11:04 am, Liang Chen < > chenliang6136@ > > wrote: > >> Hi >> Good luck! >> one question:If my system using YARN cluster, can i use Amaterasu for >> development? >> >> ------------------------------------------------------------------------------ >> We have also created a Mesos cluster you can use to test Amaterasu or use >> for development purposes >> >> Regards >> Liang >> >> 2017-03-08 4:49 GMT+08:00 Jean-Baptiste Onofré < > jb@ > >: >> >> > Hi all, >> > >> > I would like to submit a new proposal to bring Amaterasu to the Apache >> > Software Foundation incubator. >> > >> > The proposal is included below and available on the wiki: >> > >> > https://wiki.apache.org/incubator/AmaterasuProposal >> > >> > We are eager to get your comments and questions. >> > >> > Thanks ! >> > JB (on behalf of the Amaterasu community) >> > >> > = Apache Amaterasu = >> > >> > == Abstract == >> > >> > Apache Amaterasu is a framework providing continuous deployment for Big >> > Data pipelines. >> > >> > It provides the following capabilities: >> > >> > * '''Continuous integration''' tools to '''package pipelines and run >> > tests'''. >> > * A repository to store those packaged applications: the >> '''applications >> > repository'''. >> > * A repository to store the pipelines, and engine configuration (for >> > instance, location of the Spark master, etc.): per environment - the >> > '''configuration repository'''. >> > * A '''dashboard''' to monitor the pipelines. >> > * A '''DSL and integration hooks''' allowing third parties to easily >> > integrate. >> > >> > == Proposal == >> > >> > Amaterasu is a simple and powerful framework to build and dispense >> > pipelines. It aims to help data engineers and data scientists to >> compose, >> > configure, test, package, deploy and execute data pipelines written >> using >> > multiple tools, languages and frameworks. Amaterasu provides a standard >> > repo structure to package big data pipelines, a YAML based Domain >> Specific >> > Languages (DSL) for data engineers, data scientists and operations >> > engineers to manage complex pipelines throughout their entire lifecycle >> > (Dev, UAT, Prod, etc.). >> > >> > == Background == >> > >> > Amaterasu is a relatively new project that was created to deal with >> some >> > of the issues that as Consultants, we have seen recurring at different >> > client sites. Mainly the need to continuously deploy complex pipelines >> > built in multiple tools and languages. >> > Amaterasu started as a pet project and is currently being evaluated by >> a >> > couple of organizations, supported by the contributors, on a personal >> time >> > and voluntary bases. >> > >> > == Rational == >> > >> > As software engineers working on big data projects we have straggled >> for >> a >> > long time to apply the same CI/CD practices that have become the >> standard >> > in the software industry for the last few years. While some of them are >> > possible, for example Apache Spark is easy to unit test. However large >> > scale pipelines are more complex and often use data, which might be >> > un-structured as integration point, which requires heavy integration >> tests. >> > >> > To automate such tests and complex deployments, we have found the need >> to >> > often handcraft scripts and use a mixture tools, so we have decided to >> > finally build a tool we can apply in a general way and not on a project >> by >> > project basis. >> > >> > Another issue Amaterasu is trying to tackle is the Integrating between >> the >> > work of software engineers, data scientists, and sometimes operations >> > engineers. The approach Amaterasu takes to integrate between those >> three >> > schools of thought it to provide a simple YAML based DSL that provides >> a >> > simple way to integrate different pipeline written in the native tools >> for >> > each task (R, Spark in different languages, etc.). >> > >> > == Initial Goals == >> > >> > Our initial goals are to bring Amaterasu into the ASF, transition >> internal >> > engineering processes into the open, and foster a collaborative >> development >> > model according to the "Apache Way". >> > >> > In addition, we intend to continue the development of Amaterasu, add >> new >> > features as well as integrate better with other frameworks, including: >> > >> > * Apache Arrow >> > * Apache Hive >> > * Apache Drill >> > * Apache Beam >> > * Apache YARN >> > * Farther and more complete integration with Apache Spark >> > >> > Other frameworks will be evaluated after those initial goals are >> reached. >> > >> > == Current Status == >> > >> > Amaterasu is preview state but provide a large set of features. We plan >> to >> > stabilize and head to a first production ready release during the >> > incubation process. The current license is already Apache 2.0. >> > >> > === Meritocracy === >> > >> > We intend to radically expand the initial developer and user community >> by >> > running the project in accordance with the "Apache Way". Users and new >> > contributors will be treated with respect and welcomed. By >> participating >> in >> > the community and providing quality patches/support that move the >> project >> > forward, they will earn merit. They also will be encouraged to provide >> > non-code contributions (documentation, events, community management, >> etc.) >> > and will gain merit for doing so. Those with a proven support and >> quality >> > track record will be encouraged to become committers. >> > >> > === Community === >> > >> > As a relatively new project, Amaterasu has a small, but growing >> community. >> > Amaterasu is an open project, not just with it’s source code but also >> with >> > our discussions which are held openly in our slack >> > https://shintoio.slack.com which contains channels for design, tech and >> > future directions discussions. >> > >> > If Amaterasu is accepted for incubation, the primary initial goal is to >> > build a large and strong community. We are confident that Amaterasu can >> > become a key project for big data operations, which hopefully will >> create a >> > large community of users and developers. >> > >> > === Known Risks === >> > >> > Development has been sponsored mostly by a one company. For the project >> to >> > fully transition to the Apache Way governance model, development must >> shift >> > towards the meritocracy-centric model of growing a community of >> > contributors balanced with the needs for extreme stability and core >> > implementation coherency. >> > >> > === Orphaned products === >> > >> > We are fully committed on Amaterasu. A few organizations have expressed >> > their interest in using Amaterasu. >> > >> > === Inexperience with Open Source === >> > >> > We have been developing and using open source software for a long time. >> > Additionally, several ASF veterans have agreed to mentor the project >> and >> > they are listed in this proposal. The project will rely on their >> guidance >> > and collective wisdom to quickly transition the entire team of initial >> > committers towards practicing the Apache Way. >> > >> > === Reliance on Salaried Developers === >> > >> > Most of the current contributors are employed in the Big Data space. >> While >> > they might wander from their current employers, they are unlikely to >> > venture far from their core expertises and thus will continue to be >> engaged >> > with the project regardless of their current employers. >> > >> > === An Excessive Fascination with the Apache Brand === >> > >> > While we intend to leverage the Apache ‘branding’ when talking to other >> > projects as testament of our project’s ‘neutrality’, we have no plans >> for >> > making use of Apache brand in press releases nor posting billboards >> > advertising acceptance of Amaterasu into Apache Incubator. >> > >> > The main purpose in applying for Apache incubation is due to the fact >> that >> > Amaterasu is built with integration already in mind for many tools >> which >> > are Apache projects, and we see Amaterasu as an extension of these >> > projects. We hope that by being an Apache project, we can integrate >> better, >> > and collaborate more effectively with the relevant projects. As >> Amaterasu >> > matures, we see mutual benefits for all involved. >> > >> > === Initial Source === >> > >> > https://github.com/shintoio/amaterasu >> > >> > === External Dependencies === >> > >> > All external dependencies are licensed under an Apache 2.0 license or >> > Apache-compatible license. As we grow the Amaterasu community we will >> > configure our build process to require and validate all contributions >> and >> > dependencies are licensed under the Apache 2.0 license or are under an >> > Apache-compatible license. >> > >> > * Apache Spark >> > * Apache Hadoop >> > * Apache Maven (maven-core) >> > * Apache Commons >> > * Apache Log4j >> > * Apache Mesos >> > * Apache Zookeeper >> > * Apache Curator >> > * Scala >> > * Junit >> > * Py4j >> > >> > Future versions are planned to integrate with: >> > >> > * Apache YARN >> > * Apache Hive >> > * Apache Drill >> > >> > === Required Resources === >> > >> > ==== Mailing lists ==== >> > >> > * > private@.apache > (moderated subscriptions) >> > * > commits@.apache >> > * > dev@.apache >> > * > issues@.apache >> > >> > ==== Git Repository ==== >> > >> > * https://git-wip-us.apache.org/repos/asf/incubator-amaterasu.git >> > >> > ==== Issue Tracking ==== >> > >> > * JIRA Project Amaterasu >> > >> > ==== Initial Committers ===== >> > >> > * Yaniv Rodenski >> > * Jean-Baptiste Onofré >> > * Eyal Ben Ivri >> > * Karel Alfonso >> > * Kirupagaran (Kirupa) Devarajan >> > * Nadav Har Tzvi >> > >> > ==== Affiliations ==== >> > >> > * Yaniv Rodenski - Shinto >> > * Jean-Baptiste Onofré - Talend >> > * Olivier Lamy - Webtide >> > >> > ==== Sponsors ==== >> > >> > ==== Champion ==== >> > >> > * Jean-Baptiste Onofré >> > >> > ==== Mentors ==== >> > >> > * Jean-Baptiste Onofré >> > * Olivier Lamy >> > >> > ==== Sponsoring Entity ==== >> > >> > The Apache Incubator >> > >> > -- >> > Jean-Baptiste Onofré >> > > jbonofre@ >> > http://blog.nanthrax.net >> > Talend - http://www.talend.com >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: > general-unsubscribe@.apache >> > For additional commands, e-mail: > general-help@.apache >> > >> > >> >> >> -- >> Regards >> Liang >> -- View this message in context: http://apache-incubator-general.996316.n3.nabble.com/DISCUSS-Apache-Amaterasu-Incubator-Proposal-tp54036p54041.html Sent from the Apache Incubator - General mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
