Hi Guys, Looks great and sounds very complimentary to OODT. Happy that you guys mention alignment -- might be nice to expose data from an OODT file management+workflow management+resource management system as a Airavata gateway. Just a quick thought after skimming the proposal.
Good stuff! Cheers, Chris On Apr 23, 2011, at 2:53 PM, Ross Gardler wrote: > I would like to propose Airavata for entry into the Apache Incubator. > > The full proposal can be found at [1] and is copied at the end of this > mail. For those in a hurry here's a quick summary: > > Airavata is a software toolkit currently used to build science gateways > but that has a much wider potential use. It provides features to > compose, manage, execute, and monitor large scale applications and > workflows on computational resources ranging from local clusters to > national grids and computing clouds. Users can use Airavata back end > services and build gadgets to deploy in open social containers such as > Apache Rave and modify them to suite their needs. Airavata builds on > general concepts of service oriented computing, distributed messaging, > and workflow composition and orchestration. > > Airavata will provide web interfaces and scalable Service Oriented > Architecture based backend services to build or enhance Science Gateway > (see https://www.teragrid.org/web/science-gateways/) and similar > environments. Airavata will specifically focus on: > > 1. sophisticated server-side tools for registering and managing large > scale applications on computational resources. > > 2. graphical user interfaces to construct, execute, control, manage and > reuse of scientific workflows. > > 3. interfacing and interoperability with with various external (third > party) data and provenance management tools > > The project team consists of a number of existing Apache Committers and > the code comes from the same stable as some of the code donated to > Apache Rave (Incubating). > > We welcome your questions, suggestions, observations and support. > > Ross > > [1] http://wiki.apache.org/incubator/AiravataProposal > > FULL PROPOSAL TEXT > ================== > > = Airavata Proposal for Apache Incubator = > > == Abstract == > Airavata is a software toolkit currently used to build science > gateways but that has a much wider potential use. It provides features > to compose, manage, execute, and monitor large scale applications and > workflows on computational resources ranging from local clusters to > national grids and computing clouds. Users can use Airavata back end > services and build gadgets to deploy in open social containers such as > Apache Rave and modify them to suite their needs. Airavata builds on > general concepts of service oriented computing, distributed messaging, > and workflow composition and orchestration. > > > == Proposal == > > Airavata will provide web interfaces and scalable Service Oriented > Architecture based backend services to build or enhance Science > Gateway (see https://www.teragrid.org/web/science-gateways/) > and similar environments. Airavata will specifically focus on: > > 1. sophisticated server-side tools for registering and managing large > scale applications on computational resources. > 2. graphical user interfaces to construct, execute, control, manage > and reuse of scientific workflows. > 3. interfacing and interoperability with with various external (third > party) data and provenance management tools. > > == Background == > Working in close quarters with Apache Axis2 committers and inspired by > the true open source community driven software development of ASF, > Suresh Marru and Marlon Pierce have been pioneering the idea of a > Science Gateways software-based Apache project since late 2008. Many > Apache members have fostered these ideas and guided them to arrive at > this proposal. > > Currently the software is a actively used in various science > gateways. But the tools are general purpose and build upon widely used > Apache tools like Axis2, ODE engine. The core team is motivated to > expand the community and build a community welcoming both synergistic > software components and also new usage scenarios. > > It is perhaps worth noting that one of the three seed projects that > make up the Apache Rave (Incubating) project is also the product of > this same team and is derived from the same Science Gateways > community. > > == Rationale == > > The nature of computational problems has evolved from simple desktop > calculations to complex, multidisciplinary activities that require the > monitoring and analysis of remote data streams, database and web > search and large ensembles of simulations. In the academic domain > Science Gateways have emerged to address these needs and have built > software platforms that provide a community of users with the ability > to easily solve computational problems within a specific domain. The > tools developed to support these gateways are potentially of value to > any organisation needing to perform complex computations. Gateways > provide a convenient interface to the underlying infrastrucure without > the need for a deep understanding of the intricacies that > infrastructure. > > We summarize the rationale for choosing The Apache Software Foundation > (ASF) below. This is what we hope to gain from participating in the > ASF. > > 1. '''Broader impact''': our science gateway tool set is based on > Service Oriented Architecture principles, and it has always been our > goal to align our software with broader trends in the development of > software for distributed systems. Participating in the ASF provides a > concrete way to implement this idea. In particular, we have done > extensive work on the workflow systems, messaging, and application > management as Web services from the perspective of computational science > use cases (i.e., high failure rates, very long running jobs, dynamic > service creation, workflows not expressible as directed acyclic graphs, > etc). These requirements and our work to implement them have already had > direct impact on the Apache Axis 2 and Apache ODE projects. As an Apache > project, it is hoped that our community will have an enhanced > opportunity for collaboration and complementary development with Apache > Hadoop (for scientific application management), Apache QPID (for > messaging), Apache Rave (incubator - Open Social Container) and others. > It is our goal to expand our software’s usage beyond just science > gateways to the broader enterprise community. > 2. '''Sustainability''': Science gateway software development (and > cyberinfrastructure software generally) is primarily funded in the US by > the National Science Foundation (NSF), so the long term sustainability > of software across funding cycles is a longstanding problem. The NSF is > attempting to solve this problem, and its vision for sustainable > software is described here: > http://www.nsf.gov/pubs/2010/nsf10015/nsf10015.jsp. Participating in the > ASF is our project’s vision for reaching software sustainability that > underpins the NSF CF21 vision. As a successful ASF project (after > incubation), we will have created a community led, rather than funding > led, environment for the development of our sotware. This community, > through our community engagement work and adoption of meritocratic > principles, will expand beyond our current core team and existing > project collaborations. This will greatly increase the chances that our > software will continue to grow and improve beyond the participation of > any individuals. > 3. '''Maturity''': much of the software included in this proposal was > developed initially by graduate students as part of their Ph. D. work. > The Open Grid Computing Environment has devoted significant effort > (through salaried staff and volunteers from collaborating institutions) > to convert these research projects into mature, reliable, well-written, > packaged components. The code is currently hosted at SourceForge, but > we recognize the need to go beyond just the SourceForge support tools to > participate in a real community of software engineering experts. It is > our desire, through the Apache Incubator, to take our software > engineering efforts to a higher level by learning from the substantial > experience of appropraite Apache Committers. Apache mentors will provide > initial guidance, as will the attraction of additional committers from > the relevant Apache projects. > > == Initial Goals == > > * Implement a standalone version of the code base with a simple hello > world service, workflow and gadget(s) to access the examples. > * Migration of documentation and design knowledge from existing SF > project > * Re-architect Grid based security (GSI) dependencies and adopt more > general purpose security implementations. > * Make sure Cloud (including hadoop) support is more first class. > * Aim to have the first Apache release within the first 6 months > * Verify with Apache Legal that some of the more esoteric licences in > our dependencies are acceptable, or replace them as appropriate > > == Current Status == > > The proposed tools are currently hosted on SourceForge at > http://sourceforge.net/projects/ogce/ (source at > https://ogce.svn.sourceforge.net/svnroot/ogce/ogce-xbaya-gui/) and are > described at http://www.collab-ogce.org. > > == Meritocracy == > > A significant portion of initial committers are already ASF > Committers/Members, > and the entire team is well experienced with open source software > development. The existing code base has resulted from > multi-institutional collaborative projects. The developers are well > aware of the Apache way and will honor the meritocracy policy of ASF > foundation. > > == Community == > > To date our focus has been serving our immediate partners needs rather > than looking outwards in order to build a broader community with > diverse needs. Whilst the core team area likely to remain focussed on > the Science Gateways communities we are keen to welcome community > members from other disciplines. > > == Core Developers == > Our core developers consist of participants from academic, > not-for-profit and for-profit organisations. Many are already well > versed in The Apache Way. > > Amongst our initial team we have one or more committers on the > following Apache top level projects; axis, geronimo, synapse, ws, > ws-pmc, ws-woden as well as Apache Rave (Incubating). > > == Alignment == > Airavata software is built upon Apache Projects like Axis2, ODE, > Rampart, Tomcat and Maven. We will try to closely align the project > with ODE to ensure BPEL workflow compatibility. We will align with > metadata management projects like Apache OODT. Web interfaces within > the Airavata software will be synergistically developed with Apache > Rave. > > == Known Risks == > === Orphaned products === > We acknowledge the need to seek project contributions outside the current > developers. The core team actively travels and conducts workshops and > tutorials at relevant academic conferences like Supercomputing, TeraGrid, > Collaborative Technologies Systems and SciDAC. Previous experiences > have showed that these tutorials and outreach efforts will bring in > community participation. The general strategy will be to encourage > users to be active in the community and develop patches and > contribute. Also, the core developers use the Airavata software in > multiple projects with a life span ranging from 2 to 10 years, so the > risk of orphaned products is very minimal. > > Furthermore, by opening our doors to non-academic organisations > already adopting large scale computation related projects in the ASF > we hope to be able to build community beyond the proposing teams > Science Gateway interests. > > === Inexperience with Open Source === > The core team is very familiar with open source practices. The > developers include existing Apache members who have long term experience > with > the Apache Way. The OGCE project has been an active > open source project in SourceForge since November 2006. We welcome the > new directions and are well prepared to follow the Apache way. > > === Homogenous Developers === > We have a semi-distributed development environment distributed among > Indiana University and Lanka Software Foundation. We fully expect > contributions from the partnering science gateways adding to the > heterogeneous development. > > === Reliance on Salaried Developers === > The core developers are self motivated on the project and also are > funded through various federal, state and endowment research > grants. Participation in these research efforts based on Airavata > software is mostly voluntary and above and beyond the requirements of > the salaried jobs. > > The Open Gateway Computing project, from which the initial code > donation is sourced, is funded for the next 3 years and is mandated by > the funding guidelines to open source software development - > http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=1032742. We > believe in the Airavata software capabilities and its vital role in > providing sustainable middleware for Science Gateways. Nevertheless, > the core team will actively build upon Airavata software and foster > developer community outside the current core. > > === Relationships with Other Apache Products === > See “Alignment” above. Airavata is based on the concepts of Service > Oriented Architecture and all services run within Tomcat > container. The web services are based on Axis2. The orchestration of > the scientific workflows uses Orchestration Director Engine. The > software is built using Apache Maven. > > === An Excessive Fascination with the Apache Brand === > The Apache brand would certainly help promote the software suite, but > gaining the brand is not the motivation for this project. Airavata is > being proposed to Apache because of the belief in Apache’s meritocracy > model for mentored, community-driven, open source software is the best > way to develop sustainable software. See “Rational” above. Most > importantly, The Apache Software Foundation will help us create an > institution-neutral contribution venue and will help us build a > long-standing community around Airavata to sustain and improve it > beyond the span of specific, targeted research grants. > > == Documentation == > Existing documentation is available from the OGCE wiki, > http://www.collab-ogce.org/ogce/index.php/Main_Page. In addition, > there is abundance of presentation and self guided video tutorial > material. Effort will be put in to collect all this information into > meaningful documentation on the Apache websites. > > == Initial Source == > The initial source of the project is in SourceForge. The source is > available for anonymous check out from svn at > https://ogce.svn.sourceforge.net/svnroot/ogce/ogce-xbaya-gui/ > > == Source and Intellectual Property Submission Plan == > Indiana University is the current holder of Intellectual Property > rights for the software. The university has approved the code donation > and signed trustees approval, Corporate Contributor Licence Agreement > and Software Grant Agreement have been emailed to ASF secretary and > received acknowledgement. > > Specifically Indiana University will donate 4 components into Airavata > project. > > 1. XBaya Scientific Workflow Suite - includes a GUI for workflow > composition and monitoring. The composed workflow can be exported to > various workflow languages like BPEL, SCUFL, Condor DAG, Jython and > Java. The defacto workflow enacting engine used is Apache ODE. > 2. GFac - an application wrapper service that can be used to wrap > command line-driven science applications and make them into robust, > network- accessible services. This component is build on Axis2 web > service stack. > 3. XRegistry - a registry service for storing deployment information > about wrapped application services and constructed workflows. > 4. WS-Messenger - a “publish-subscribe” based message broker > implemented on top of Apache Axis2 web services stack. It implements the > WS-Eventing and WS-Notifications specifications and incorporates a > message box component that facilities communications with clients behind > firewalls and overcomes network glitches. > > == External Dependencies == > > Following the guideline -http://www.apache.org/legal/resolved.html, the > following are the dependent software and all of them are in binary > format in java archive (jar files). > > * CDDL license - Javax activation, JSR311, Portlet-API, Servlet-API > * Apache V2: cog-jglobus, globus, caster, gridsphere, Woodstox, > xmpp, xsul, sigiri, atomixmiser, weps-beans. > * BSD: puretls, > * MIT: bcporv, hsqldb, dom4j, slf4j > * PSFL: Jython > * GPL 2.0: mysql-connector-java > * Other: > * cryptix32, cryptix-asn1 (http://www.cryptix.org/LICENSE.TXT) > * backport (public domain) > * jaxen (http://jaxen.codehaus.org/license.html) > > Licence incompatibilities (GPL) will be resolved during incubation. > > == Cryptography == > The software does not implement any cryptographic algorithms. However, > to perform secured messaging and data movement and SSL communications, > the software depends upon third party security libraries. These > external libraries depend in turn on Java Security, Puretls, Cryptix > and Bounce Castle libraries. Apache Cryptographic steps will be > followed to register the use of these libraries. > > == Required Resources == > > === Mailing lists === > 1. airavata-dev > 2. airavata-commits > 3. airavata-private > > === Subversion Directory === > https://svn.apache.org/repos/asf/incubator/airavata > > === Issue Tracking === > We intend to make use of Jira for issue tracking. Proposed key: AIRAVATA > > === Other Resources === > > We intend to manage our website using the Apache CMS. > > == Initial Committers == > Names of initial committers with affiliation and current ASF status: > > || '''Name''' || '''Email''' || '''Affiliation''' > || '''ICLA''' || '''ASF Status''' || '''Apache Id''' || > || Suresh Marru || sma...@cs.indiana.edu || Indiana University > || On File || Apache Commiter || smarru || > || Marlon Pierce || mpie...@cs.indiana.edu || Indiana University > || On File || Apache Commiter || mpierce || > || Srinath Perera || hemap...@apache.org || Lanka Software > Foundation || On File || Apache Member || hemapani || > || Aleksander Slominski || aslom at us.ibm.com || IBM > || On File || Apache Member || aslom || > || Raminderjeet Singh || rami...@indiana.edu || Indiana University > || On File || Apache Commiter || raminder || > || Archit Kulshrestha || akuls...@indiana.edu || Indiana University > || On File || N/A || N/A || > || Chathura Herath || chath...@apache.org || Indiana University > || On File || Apache Commiter || chathura || > || Eran Chinthaka || chinth...@apache.org || Indiana University > || On File || Apache Member || chinthaka || > || Thilina Gunaratne || thil...@apache.org || Indiana University > || On File || Apache Commiter || thilina || > || Wathsala Vithanage || waths...@opensource.lk || Lanka Software > Foundation || On File || N/A || N/A || > > All the parties are affiliated with companies and organizations that > are familiar with the development of open source. We expect that the > amount of volunteer work will increase, and more developers will come > on board. > > == Champion == > Ross Gardler, Apache Software Foundation > > == Nominated Mentors == > * Ross Gardler, Member, Apache Software Foundation > * Alek Slominski, Member, Apache Software Foundation > * Ate Douma, Member, Apache Software Foundation > * Sanjiva Weerawarna, Member, Apache Software Foundation > * Paul Fremantle, Member, Apache Software Foundation > > == Sponsoring Entity == > Apache Incubator Project. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
