Hi Folks, Having secured a mentorship team consisting of the following IPMC Members, I am happy to open a formal VOTE thread on accepting the Science Data Analytics Platform (SDAP) into Apache Incubator.
- Lewis John McGibbney (lewi...@apache.org) - Raphael Bircher (bircher at apace dot org) - Suneel Marthi (smarthi at apache dot org) Thank you to both Raphael and Suneel for coming forward. :) The VOTE will be open for at least 72 hours. [ ] +1 Accept Science Data Analytics Platform (SDAP) into Apache Incubator [ ] +/-0 ... just because [ ] -1 Do NOT Accept Science Data Analytics Platform (SDAP) into Apache Incubator... because Thanks in advance to all participants. Lewis P.S. Here is a binding +1 from me On Wed, Oct 11, 2017 at 11:22 AM, lewis john mcgibbney <lewi...@apache.org> wrote: > Hi Folks, > I would like to open a DISCUSS thread on the topic of accepting the > Science Data Analytics Platform (SDAP) <https://wiki.apache.org/ > incubator/SDAPProposal> Project into the Incubator. > I am CC'ing Thomas Huang from NASA JPL who I have been working with to > build community around a kick-ass set of software projects under the SDAP > umbrella. > At this stage we would very much appreciate critical feedback from general@ > community. We are also open to mentors who may have an interest in the > project proposal. > The proposal is pasted below. > Thanks in advance, > Lewis > > = Abstract = > The Science Data Analytics Platform (SDAP) establishes an integrated data > analytic center for Big Science problems. It focuses on technology > integration, advancement and maturity. > > = Proposal = > SDAP currently represents a collaboration between NASA Jet Propulsion > Laboratory (JPL), Florida State University (FSU), the National Center for > Atmospheric Research (NCAR), and George Mason University (GMU). SDAP brings > together a number of big data technologies including a NASA funded > OceanXtremes (Anomaly detection and ocean science), NEXUS (Deep data > analytic platform), DOMS (Distributed in-situ to satellite matchup), MUDROD > (Search relevancy and discovery) and VQSS (Virtualized Quality Screening > Service) under a single umbrella. Within the original Incubator proposal, > VQSS will not be included however it is anticipated that a future source > code donation will cover VQSS. > > = Background and Rationale = > SDAP is a technology software solution currently geared to better enable > scientists involved in advancing the study of the Earth's physical > oceanography. With increasing global temperature, warming of the ocean, and > melting ice sheets and glaciers, the impacts can be observed from changes > in anomalous ocean temperature and circulation patterns, to increasing > extreme weather events and stronger/more frequent hurricanes, sea level > rise and storm surges affecting coastlines, and may involve drastic changes > and shifts in marine ecosystems. Ocean science communities are relying on > data distributed through data centers such as the JPL's Physical > Oceanographic Data Active Archive Center (PO.DAAC) to conduct their > research. In typical investigations, oceanographers follow a traditional > workflow for using datasets: search, evaluate, download, and apply tools > and algorithms to look for trends. While this workflow has been working > very well historically for the oceanographic community, it cannot scale if > the research involves massive amount of data. NASA's Surface Water and > Ocean Topography (SWOT) mission, scheduled to launch in April of 2021, is > expected to generate over 20PB data for a nominal 3-year mission. This will > challenge all existing NASA Earth Science data archival/distribution > paradigms. It will no longer be feasible for Earth scientists to download > and analyze such volumes of data. SDAP was therefore developed primarily as > a Web-service platform for big ocean data science at the PO.DAAC with open > source solutions used to enable fast analysis of oceanographic data. SDAP > has been developed collaboratively between JPL, FSU, NCAR, and GMU and is > rapidly maturing to become the generic platform for the next generation of > big science data solutions. The platform is an orchestration of several > previously funded NASA big ocean data solutions using cloud technology, > which include data analysis (NEXUS), anomaly detection (OceanXtremes), > matchup (DOMS), subsetting, discovery (MUDROD), and visualization (VQSS). > SDAP will enable web-accessible, fast data analysis directly on huge > scientific data archives to minimize data movement and provide access, > including subset, only to the relevant data. > > = Science Data Analytics Platform Project Overview = > SDAP consists of several loosely coupled, independently functioning > sub-projects. The graphic below displays an overview of how these > sub-projects fuse together. N.B., although the graphic uses terminology > relating to OceanWorks, essentially the SDAP architecture is identical. > > {{attachment:sdap.png}} > > == OceanXtremes == > Oceanographic Data-Intensive Anomaly Detection and Analysis Portal. An > application that allows you to view imagery and perform analysis on sea > level rise data. > > '''Objective''' > Develop an anomaly detection system which identifies items, events or > observations which do not conform to an expected pattern. > * Mature and test domain-specific, multi-scale anomaly and feature > detection algorithms. > * Identify unexpected correlations between key measured variables. > > Demonstrate value of technologies in this service: > * Adapted Map-Reduce data mining. > * Algorithm profiling service. > * Shared discovery and exploration search tools. > * Automatic notification of events of interest. > > == NEXUS == > NEXUS is an emerging technology developed at JPL > * A Cloud-based/Cluster-based data platform that performs scalable > handling of observational parameters analysis designed to scale horizontally > * Leveraging high-performance indexed, temporal, and geospatial search > solution > * Breaks data products into small chunks and stores them in a Cloud-based > data store > > ''Data Volumes Exploding'' > * SWOT mission is coming > * File I/O is slow > > ''Scalable Store & Compute is Available'' > * NoSQL cluster databases > * Parallel compute, in-memory map-reduce > * Bring Compute to Highly-Accessible Data (using Hybrid Cloud) > > ''Pre-Chunk and Summarize Key Variables'' > * Easy statistics instantly (milliseconds) > * Harder statistics on-demand (in seconds) > * Visualize original data (layers) on a map quickly > > == DOMS == > The Distributed Oceanographic Match-Up Service > DOMS is designed to reconcile satellite and in situ datasets in support of > NASA's Earth Science mission. The service will provide a mechanism for > users to input a series of geospatial references for satellite observations > and receive the in situ observations that are matched to the satellite data > within a selectable temporal and spatial domain. DOMS includes several > characteristic in situ and satellite observation datasets - with an initial > focus on salinity, sea temperature, and winds. DOMS will be used by the > marine and satellite research communities to support a range of activities > and several use cases will be described. The service is designed to provide > a community-accessible tool that dynamically delivers matched data and > allows the scientist to only work with the subset of data where the matches > exist. > > == MUDROD == > Mining and Utilizing Dataset Relevancy from Oceanographic Datasets to > Improve Data Discovery and Access > Data discovery accuracy is a challenging topic for both Earth science and > other domains. It is especially true for scientific data sets that are not > as popular as Amazon or Google data. MUDROD is focused on mining oceanic > knowledge from the PO.DAAC user log files to improve the end user data > discovery experience at PO.DAAC. There are three steps in the research: a) > the oceanographic semantics were extracted from three resources of SWEET, > GCMD ontology, and the keywords used by end users for searching PO.DAAC > datasets, b) mining the linkage among different vocabularies based on user > data discvoery sessions, and c) build the linkage among vocabularies based > on a comprehensive approach by considering domain de facto standard, e.g., > SWEET and GCMD, and the knowledge mined from the log files. The semantics > is used to improve data discovery for ranking results, navigating among > vocabularies, and recommending data based on user searchers. > > = Current Status = > All components of SDAP were originally designed and developed under grants > from the NASA-funded Advanced Information Systems and Technologies (AIST) > program. The initiative to bring them the components together under the > SDAP umbrella was granted through an AIST-funded follow-on grant which will > run for another ~18 or so months. > Currently no projects have made official releases so outside of community > building, this will be our primary Incubating goal. All SDAP source code is > currently publicly available and licensed under the ALv2.0. > > = Meritocracy = > The current developers are familiar with meritocratic open source > development at Apache. The SDAP team consumes Apache products heavily with > members being part of several Apache user communities. SDAP itself has > critical dependencies upon Apache products. Lewis McGibbney (JPL employee), > a Member of the ASF and V.P. of Apache Any23, Gora PMC Nutch, Tika, OODT, > OCW, etc., is championing the effort to bring SDAP into and through the > Apache Incubator and has been evangelizing the Apache Way to the current > SDAP contributors such that the meritocratic process is well understood and > followed. Apache was chosen specifically because we want to encourage this > style of community development for the project and for it to sustain SDAP > forward to become the generic platform for the next generation of big > science data solutions > > = Community = > The SDAP project is a fairly new effort and our community is not yet > fully/firmly established. Initial committers comprising the SDAP roster > have only recently fully come together as a unified team however there is a > large degree of synergy between constituent members at JPL, FSU, NCAR, and > GMU. Therefore, community building and publicity continues to be a major > thrust. With the activity and exposure regularly attained by several > community members, we hope to grow the SDAP presence in and across several > (scientific) forums. The SDAP technology is generating interest within > communities such as the Earth Science Information Partnership (ESIP), > American Geophysical Union (AGU) and plethora or science meetings around > the globe. This in effect, we hope, will further contribute towards the > possibility of SDAP being used across Government Agencies such as NASA, > NOAA, USGS, EPA, DOI, etc. as well as by researchers and students in > academic institutions around the globe. > During incubation, we will explicitly seek to increase our adoption, with > SDAP already being featured on the agenda for several high profile globally > significant scientific conferences and meetings. > > = Core Developers = > The current set of core developers is relatively small, including > full-time and students from across JPL, FSU, NCAR, and GMU. Initial > community management and participation will be distributed across the > entire team, most of which have been involved with the constituent projects > for <2 years. > > = Alignment = > All SDAP code is licensed under Apache v2.0. > > = Known Risks = > > == Orphaned products == > There are currently no orphaned products. Each component of SDAP has > dedicated personnel leading and participating in its ongoing development. > Additionally, there is substantial collaboration between projects > facilitated by regular project meetings which are specific the the initial > member entities and focused on advancing physical oceanographic science. > > == Inexperience with Open Source == > JPL (in particular Lewis McGibbney) has been part of several efforts to > transition to and grow projects communities at Apache e.g. Apache OODT, > Apache Open Climate Workbench, Apache Joshua (Incubating), Apache SensSoft > (Incubating), Apache DRAT (Incubating). Most of the code developed under > the SDAP umbrella was and is open source prior to the Incubator effort so > we are well familiarized with the nuances of open source software. > > = Relationships with Other Apache Products = > SDAP has strong dependency upon a number of high profile and smaller > profile Apache products. Examples can be seen in the breakdown of External > Dependencies. As we continue to grow SDAP within the Incubator, we will > make efforts to share community stories, software advancements and possible > improvements in our use of our Apache dependencies back to those project > communities. > > = Developers = > The SDAP project and hence developers is currently funded through a NASA > AIST follow-on grant with funding secured for the next ~18 months. There > are currently no 100% time dedicated developers, however, the same core > team that does work currently will continue to work on the project > throughout the next current funding period and after. There is currently no > business strategy aligned with SDAP however it is perceived that future, > yet unsecured funding may by directed to further feature advancement and > project evangelism. > > = Documentation = > Documentation is currently available in a number of locations e.g. Github > wiki, Github pages, etc. with each repository under the oceanworks-aist > Github Org maintaining documentation available through wiki’s attached to > the repositories. Additionally, most of the SDAP sub-projects have been > extensively documented within plethora of formal academic publications > across several academic communities. It would be our intention, certainly > atleast to unify the Github wiki ad Github pages documentation most likely > to make up the sdap.apache.org Website content. > > = Initial Source = > Current source resides in several locations Github: > * https://github.com/dataplumber/nexus (NEXUS, OceanXtremes, DOMS) > * https://github.com/dataplumber/edge (EDGE) > * https://github.com/aist-oceanworks/mudrod (MUDROD) > * https://bitbucket.org/coaps_mdc/doms/src (DOMS) > > = External Dependencies = > Each component of the Science Data Analytics Platform has its own > dependencies. Documentation will be available for integrating them. > > == MUDROD == > '''Core''' > com.google.code.gson gson 2.5 compile > jar false > org.jdom jdom 2.0.2 compile > jar false > org.elasticsearch elasticsearch 5.2.0 compile > jar false > org.elasticsearch elasticsearch-spark-20_2.11 5.2.0 compile > jar false > joda-time joda-time 2.9.4 compile > jar false > com.carrotsearch hppc 0.7.1 compile > jar false > org.apache.spark spark-core_2.11 2.1.0 compile > jar false > org.apache.spark spark-sql_2.11 2.1.0 compile > jar false > org.apache.spark spark-mllib_2.11 2.1.0 compile > jar false > org.scala-lang scala-library 2.11.8 compile > jar false > org.codehaus.jettison jettison 1.3.8 compile > jar false > commons-cli commons-cli 1.2 compile > jar false > net.sf.opencsv opencsv 2.3 compile > jar false > org.apache.jena jena-core 3.3.0 compile > jar false > junit junit 4.12 test > jar false > > '''Service''' > gov.nasa.jpl.mudrod mudrod-core 0.0.1-SNAPSHOT compile > jar false > javax.servlet javax.servlet-api 3.1.0 provided > jar false > com.google.code.gson gson 2.5 compile > jar false > > '''Web''' > * AngularJS - MIT License > * BootstrapJS - MIT License > * jQueryJS - MIT License > * Underscore JS - MIT License > > == DOMS == > * Apache Solr version 5.5.1http://lucene.apache.org/solr/ > * EDGE https://github.com/dataplumber/edge > * NetCDF4 http://unidata.github.io/netcdf4-python/ > * Python 3.5 (NOTE: only partial support for py2.7) > > Non stdlib Python dependencies: > * Jinja2==2.9.5 > * python-dateutil==2.6.0 > * cython==0.25.2 > * numpy==1.12.0 > * scipy==0.18.1 > * netCDF4==1.2.7 > * solrpy3 > * siphon==0.4.0 > * neo4j-driver==1.1.0 > * matplotlib==2.0.0 > * requests==2.13.0 > * shapely==1.5.17 > * flask==0.12 > * networkx==1.11 > * pyproj==1.9.5.1 > * blist==1.3.6 > > == NEXUS == > '''Analysis''' > * https://github.com/dataplumber/nexus/blob/master/ > analysis/package-list.txt > * https://github.com/dataplumber/nexus/blob/master/ > analysis/requirements.txt > > '''Client''' > * https://github.com/dataplumber/nexus/blob/master/ > client/requirements.txt > > '''Climatology''' > * matplotlib > * numpy > * netCDF4 > * pathos (https://pypi.python.org/pypi/pathos) > > '''Data-access''' > * https://github.com/dataplumber/nexus/blob/master/ > data-access/requirements.txt > > '''Nexus-ingest''' > ''Dataset-tiler'' > * https://github.com/dataplumber/nexus/tree/master/ > nexus-ingest/dataset-tiler/build/reports > > ''developer-box'' > * Just a collection of scripts/vagrant file used to stand up a developer > instance of nexus ingestion. No dependencies to report > > ''Groovy-scripts'' > * Collection of Groovy scripts that can be used as part of data > ingestion. They only rely on the standard Groovy library and the > ‘nexus-messages’ project > > ''Nexus-messages'' > * https://github.com/dataplumber/nexus/tree/master/ > nexus-ingest/nexus-messages/build/reports > > ''nexus-sink'' > * https://github.com/dataplumber/nexus/tree/master/ > nexus-ingest/nexus-sink/build/reports > > ''nexus-xd-python-modules'' > * https://github.com/dataplumber/nexus/blob/master/ > nexus-ingest/nexus-xd-python-modules/package-list.txt > * https://github.com/dataplumber/nexus/blob/master/ > nexus-ingest/nexus-xd-python-modules/requirements.txt > > ''spring-xd-python'' > * only python standard libraries are used > > ''tcp-shell'' > * https://github.com/dataplumber/nexus/tree/master/ > nexus-ingest/tcp-shell/build/reports > > '''tools/deletebyquery''' > * https://github.com/dataplumber/nexus/blob/master/tools/deletebyquery/ > requirements.txt > > = Required Resources = > Mailing Lists > * priv...@sdap.incubator.apache.org > * d...@sdap.incubator.apache.org > * comm...@sdap.incubator.apache.org > > Git Repos > * https://git-wip-us.apache.org/repos/asf/incubator-nexus.git > * https://git-wip-us.apache.org/repos/asf/incubator-doms.git > * https://git-wip-us.apache.org/repos/asf/incubator-mudrod.git > > Issue Tracking > * JIRA Science Data Analytics Platform (SDAP) > > Continuous Integration > * Jenkins builds on https://builds.apache.org/ > > Web > * http://sdap.incubator.apache.org/ > * wiki at http://cwiki.apache.org > > = Initial Committers = > The following is a list of the planned initial Apache committers (the > active subset of the committers for the current repository on Github). > * Lewis John McGibbney (lewi...@apache.org) > * Vardis M. Tsontos (vardis.m.tson...@jpl.nasa.gov) > * Joseph C. Jacob (joseph.c.ja...@jpl.nasa.gov) > * Ed Armstrong (edward.m.armstr...@jpl.nasa.gov) > * Frank Greguska (gregu...@jpl.nasa.gov) > * Brian Wilson (brian.wil...@jpl.nasa.gov) > * Chaowe Phil Yang (cya...@gmu.edu) > * Yongyao Jiang (yjia...@gmu.edu) > * Yun Li (yl...@gmu.edu) > * Shawn R. Smith (sm...@coaps.fsu.edu) > * Jocelyn Elya (je...@coaps.fsu.edu) > * Mark Bourassa (boura...@coaps.fsu.edu) > * Thomas Cram (tc...@ucar.edu) > * Thomas Huang (thomas.hu...@jpl.nasa.gov) > * Steven Worley (wor...@ucar.edu) > * Zaihua Ji (z...@ucar.edu) > > = Affiliations = > NASA JPL > * Lewis John McGibbney (lewi...@apache.org) > * Vardis M. Tsontos (vardis.m.tson...@jpl.nasa.gov) > * Joseph C. Jacob (joseph.c.ja...@jpl.nasa.gov) > * Ed Armstrong (edward.m.armstr...@jpl.nasa.gov) > * Frank Greguska (gregu...@jpl.nasa.gov) > * Thomas Huang (thomas.hu...@jpl.nasa.gov) > * Brian Wilson (brian.wil...@jpl.nasa.gov) > > George Mason University > * Chaowe Phil Yang (cya...@gmu.edu) > * Yongyao Jiang (yjia...@gmu.edu) > * Yun Li (yl...@gmu.edu) > > Center for Ocean-Atmospheric Prediction Studies, Florida State University > * Shawn R. Smith (sm...@coaps.fsu.edu) > * Jocelyn Elya (je...@coaps.fsu.edu) > * Mark Bourassa (boura...@coaps.fsu.edu) > > Computational Information Systems Laboratory (CISL) / National Center for > Atmospheric Research (NCAR) > * Thomas Cram (tc...@ucar.edu) > * Zaihua Ji (z...@ucar.edu) > * Steven Worley (wor...@ucar.edu) > > = Sponsors = > > = Champion = > * Lewis McGibbney (NASA/JPL) > > = Nominated Mentors = > * TBD > * TBD > * TBD > > = Sponsoring Entity = > The Apache Incubator > > > -- > http://home.apache.org/~lewismc/ > @hectorMcSpector > http://www.linkedin.com/in/lmcgibbney > -- http://home.apache.org/~lewismc/ @hectorMcSpector http://www.linkedin.com/in/lmcgibbney
