+1 Happy to help out if the vote passes.
On Tue, Oct 17, 2017 at 10:07 PM, Madhawa Kasun Gunasekara < madhaw...@gmail.com> wrote: > Here is my +1 > > Thanks, > Madhawa > > Madhawa > > On Tue, Oct 17, 2017 at 4:04 PM, lewis john mcgibbney <lewi...@apache.org> > wrote: > > > Hi Folks, > > Having secured a mentorship team consisting of the following IPMC > Members, > > I am happy to open a formal VOTE thread on accepting the Science Data > > Analytics Platform (SDAP) into Apache Incubator. > > > > - Lewis John McGibbney (lewi...@apache.org) > > - Raphael Bircher (bircher at apace dot org) > > - Suneel Marthi (smarthi at apache dot org) > > > > Thank you to both Raphael and Suneel for coming forward. :) > > The VOTE will be open for at least 72 hours. > > > > [ ] +1 Accept Science Data Analytics Platform (SDAP) into Apache > Incubator > > [ ] +/-0 ... just because > > [ ] -1 Do NOT Accept Science Data Analytics Platform (SDAP) into Apache > > Incubator... because > > > > Thanks in advance to all participants. > > Lewis > > > > P.S. Here is a binding +1 from me > > > > On Wed, Oct 11, 2017 at 11:22 AM, lewis john mcgibbney < > lewi...@apache.org > > > > > wrote: > > > > > Hi Folks, > > > I would like to open a DISCUSS thread on the topic of accepting the > > > Science Data Analytics Platform (SDAP) <https://wiki.apache.org/ > > > incubator/SDAPProposal> Project into the Incubator. > > > I am CC'ing Thomas Huang from NASA JPL who I have been working with to > > > build community around a kick-ass set of software projects under the > SDAP > > > umbrella. > > > At this stage we would very much appreciate critical feedback from > > general@ > > > community. We are also open to mentors who may have an interest in the > > > project proposal. > > > The proposal is pasted below. > > > Thanks in advance, > > > Lewis > > > > > > = Abstract = > > > The Science Data Analytics Platform (SDAP) establishes an integrated > data > > > analytic center for Big Science problems. It focuses on technology > > > integration, advancement and maturity. > > > > > > = Proposal = > > > SDAP currently represents a collaboration between NASA Jet Propulsion > > > Laboratory (JPL), Florida State University (FSU), the National Center > for > > > Atmospheric Research (NCAR), and George Mason University (GMU). SDAP > > brings > > > together a number of big data technologies including a NASA funded > > > OceanXtremes (Anomaly detection and ocean science), NEXUS (Deep data > > > analytic platform), DOMS (Distributed in-situ to satellite matchup), > > MUDROD > > > (Search relevancy and discovery) and VQSS (Virtualized Quality > Screening > > > Service) under a single umbrella. Within the original Incubator > proposal, > > > VQSS will not be included however it is anticipated that a future > source > > > code donation will cover VQSS. > > > > > > = Background and Rationale = > > > SDAP is a technology software solution currently geared to better > enable > > > scientists involved in advancing the study of the Earth's physical > > > oceanography. With increasing global temperature, warming of the ocean, > > and > > > melting ice sheets and glaciers, the impacts can be observed from > changes > > > in anomalous ocean temperature and circulation patterns, to increasing > > > extreme weather events and stronger/more frequent hurricanes, sea level > > > rise and storm surges affecting coastlines, and may involve drastic > > changes > > > and shifts in marine ecosystems. Ocean science communities are relying > on > > > data distributed through data centers such as the JPL's Physical > > > Oceanographic Data Active Archive Center (PO.DAAC) to conduct their > > > research. In typical investigations, oceanographers follow a > traditional > > > workflow for using datasets: search, evaluate, download, and apply > tools > > > and algorithms to look for trends. While this workflow has been working > > > very well historically for the oceanographic community, it cannot scale > > if > > > the research involves massive amount of data. NASA's Surface Water and > > > Ocean Topography (SWOT) mission, scheduled to launch in April of 2021, > is > > > expected to generate over 20PB data for a nominal 3-year mission. This > > will > > > challenge all existing NASA Earth Science data archival/distribution > > > paradigms. It will no longer be feasible for Earth scientists to > download > > > and analyze such volumes of data. SDAP was therefore developed > primarily > > as > > > a Web-service platform for big ocean data science at the PO.DAAC with > > open > > > source solutions used to enable fast analysis of oceanographic data. > SDAP > > > has been developed collaboratively between JPL, FSU, NCAR, and GMU and > is > > > rapidly maturing to become the generic platform for the next generation > > of > > > big science data solutions. The platform is an orchestration of several > > > previously funded NASA big ocean data solutions using cloud technology, > > > which include data analysis (NEXUS), anomaly detection (OceanXtremes), > > > matchup (DOMS), subsetting, discovery (MUDROD), and visualization > (VQSS). > > > SDAP will enable web-accessible, fast data analysis directly on huge > > > scientific data archives to minimize data movement and provide access, > > > including subset, only to the relevant data. > > > > > > = Science Data Analytics Platform Project Overview = > > > SDAP consists of several loosely coupled, independently functioning > > > sub-projects. The graphic below displays an overview of how these > > > sub-projects fuse together. N.B., although the graphic uses terminology > > > relating to OceanWorks, essentially the SDAP architecture is identical. > > > > > > {{attachment:sdap.png}} > > > > > > == OceanXtremes == > > > Oceanographic Data-Intensive Anomaly Detection and Analysis Portal. An > > > application that allows you to view imagery and perform analysis on sea > > > level rise data. > > > > > > '''Objective''' > > > Develop an anomaly detection system which identifies items, events or > > > observations which do not conform to an expected pattern. > > > * Mature and test domain-specific, multi-scale anomaly and feature > > > detection algorithms. > > > * Identify unexpected correlations between key measured variables. > > > > > > Demonstrate value of technologies in this service: > > > * Adapted Map-Reduce data mining. > > > * Algorithm profiling service. > > > * Shared discovery and exploration search tools. > > > * Automatic notification of events of interest. > > > > > > == NEXUS == > > > NEXUS is an emerging technology developed at JPL > > > * A Cloud-based/Cluster-based data platform that performs scalable > > > handling of observational parameters analysis designed to scale > > horizontally > > > * Leveraging high-performance indexed, temporal, and geospatial search > > > solution > > > * Breaks data products into small chunks and stores them in a > > Cloud-based > > > data store > > > > > > ''Data Volumes Exploding'' > > > * SWOT mission is coming > > > * File I/O is slow > > > > > > ''Scalable Store & Compute is Available'' > > > * NoSQL cluster databases > > > * Parallel compute, in-memory map-reduce > > > * Bring Compute to Highly-Accessible Data (using Hybrid Cloud) > > > > > > ''Pre-Chunk and Summarize Key Variables'' > > > * Easy statistics instantly (milliseconds) > > > * Harder statistics on-demand (in seconds) > > > * Visualize original data (layers) on a map quickly > > > > > > == DOMS == > > > The Distributed Oceanographic Match-Up Service > > > DOMS is designed to reconcile satellite and in situ datasets in support > > of > > > NASA's Earth Science mission. The service will provide a mechanism for > > > users to input a series of geospatial references for satellite > > observations > > > and receive the in situ observations that are matched to the satellite > > data > > > within a selectable temporal and spatial domain. DOMS includes several > > > characteristic in situ and satellite observation datasets - with an > > initial > > > focus on salinity, sea temperature, and winds. DOMS will be used by the > > > marine and satellite research communities to support a range of > > activities > > > and several use cases will be described. The service is designed to > > provide > > > a community-accessible tool that dynamically delivers matched data and > > > allows the scientist to only work with the subset of data where the > > matches > > > exist. > > > > > > == MUDROD == > > > Mining and Utilizing Dataset Relevancy from Oceanographic Datasets to > > > Improve Data Discovery and Access > > > Data discovery accuracy is a challenging topic for both Earth science > and > > > other domains. It is especially true for scientific data sets that are > > not > > > as popular as Amazon or Google data. MUDROD is focused on mining > oceanic > > > knowledge from the PO.DAAC user log files to improve the end user data > > > discovery experience at PO.DAAC. There are three steps in the research: > > a) > > > the oceanographic semantics were extracted from three resources of > SWEET, > > > GCMD ontology, and the keywords used by end users for searching PO.DAAC > > > datasets, b) mining the linkage among different vocabularies based on > > user > > > data discvoery sessions, and c) build the linkage among vocabularies > > based > > > on a comprehensive approach by considering domain de facto standard, > > e.g., > > > SWEET and GCMD, and the knowledge mined from the log files. The > semantics > > > is used to improve data discovery for ranking results, navigating among > > > vocabularies, and recommending data based on user searchers. > > > > > > = Current Status = > > > All components of SDAP were originally designed and developed under > > grants > > > from the NASA-funded Advanced Information Systems and Technologies > (AIST) > > > program. The initiative to bring them the components together under the > > > SDAP umbrella was granted through an AIST-funded follow-on grant which > > will > > > run for another ~18 or so months. > > > Currently no projects have made official releases so outside of > community > > > building, this will be our primary Incubating goal. All SDAP source > code > > is > > > currently publicly available and licensed under the ALv2.0. > > > > > > = Meritocracy = > > > The current developers are familiar with meritocratic open source > > > development at Apache. The SDAP team consumes Apache products heavily > > with > > > members being part of several Apache user communities. SDAP itself has > > > critical dependencies upon Apache products. Lewis McGibbney (JPL > > employee), > > > a Member of the ASF and V.P. of Apache Any23, Gora PMC Nutch, Tika, > OODT, > > > OCW, etc., is championing the effort to bring SDAP into and through the > > > Apache Incubator and has been evangelizing the Apache Way to the > current > > > SDAP contributors such that the meritocratic process is well understood > > and > > > followed. Apache was chosen specifically because we want to encourage > > this > > > style of community development for the project and for it to sustain > SDAP > > > forward to become the generic platform for the next generation of big > > > science data solutions > > > > > > = Community = > > > The SDAP project is a fairly new effort and our community is not yet > > > fully/firmly established. Initial committers comprising the SDAP roster > > > have only recently fully come together as a unified team however there > > is a > > > large degree of synergy between constituent members at JPL, FSU, NCAR, > > and > > > GMU. Therefore, community building and publicity continues to be a > major > > > thrust. With the activity and exposure regularly attained by several > > > community members, we hope to grow the SDAP presence in and across > > several > > > (scientific) forums. The SDAP technology is generating interest within > > > communities such as the Earth Science Information Partnership (ESIP), > > > American Geophysical Union (AGU) and plethora or science meetings > around > > > the globe. This in effect, we hope, will further contribute towards the > > > possibility of SDAP being used across Government Agencies such as NASA, > > > NOAA, USGS, EPA, DOI, etc. as well as by researchers and students in > > > academic institutions around the globe. > > > During incubation, we will explicitly seek to increase our adoption, > with > > > SDAP already being featured on the agenda for several high profile > > globally > > > significant scientific conferences and meetings. > > > > > > = Core Developers = > > > The current set of core developers is relatively small, including > > > full-time and students from across JPL, FSU, NCAR, and GMU. Initial > > > community management and participation will be distributed across the > > > entire team, most of which have been involved with the constituent > > projects > > > for <2 years. > > > > > > = Alignment = > > > All SDAP code is licensed under Apache v2.0. > > > > > > = Known Risks = > > > > > > == Orphaned products == > > > There are currently no orphaned products. Each component of SDAP has > > > dedicated personnel leading and participating in its ongoing > development. > > > Additionally, there is substantial collaboration between projects > > > facilitated by regular project meetings which are specific the the > > initial > > > member entities and focused on advancing physical oceanographic > science. > > > > > > == Inexperience with Open Source == > > > JPL (in particular Lewis McGibbney) has been part of several efforts to > > > transition to and grow projects communities at Apache e.g. Apache OODT, > > > Apache Open Climate Workbench, Apache Joshua (Incubating), Apache > > SensSoft > > > (Incubating), Apache DRAT (Incubating). Most of the code developed > under > > > the SDAP umbrella was and is open source prior to the Incubator effort > so > > > we are well familiarized with the nuances of open source software. > > > > > > = Relationships with Other Apache Products = > > > SDAP has strong dependency upon a number of high profile and smaller > > > profile Apache products. Examples can be seen in the breakdown of > > External > > > Dependencies. As we continue to grow SDAP within the Incubator, we will > > > make efforts to share community stories, software advancements and > > possible > > > improvements in our use of our Apache dependencies back to those > project > > > communities. > > > > > > = Developers = > > > The SDAP project and hence developers is currently funded through a > NASA > > > AIST follow-on grant with funding secured for the next ~18 months. > There > > > are currently no 100% time dedicated developers, however, the same core > > > team that does work currently will continue to work on the project > > > throughout the next current funding period and after. There is > currently > > no > > > business strategy aligned with SDAP however it is perceived that > future, > > > yet unsecured funding may by directed to further feature advancement > and > > > project evangelism. > > > > > > = Documentation = > > > Documentation is currently available in a number of locations e.g. > Github > > > wiki, Github pages, etc. with each repository under the oceanworks-aist > > > Github Org maintaining documentation available through wiki’s attached > to > > > the repositories. Additionally, most of the SDAP sub-projects have been > > > extensively documented within plethora of formal academic publications > > > across several academic communities. It would be our intention, > certainly > > > atleast to unify the Github wiki ad Github pages documentation most > > likely > > > to make up the sdap.apache.org Website content. > > > > > > = Initial Source = > > > Current source resides in several locations Github: > > > * https://github.com/dataplumber/nexus (NEXUS, OceanXtremes, DOMS) > > > * https://github.com/dataplumber/edge (EDGE) > > > * https://github.com/aist-oceanworks/mudrod (MUDROD) > > > * https://bitbucket.org/coaps_mdc/doms/src (DOMS) > > > > > > = External Dependencies = > > > Each component of the Science Data Analytics Platform has its own > > > dependencies. Documentation will be available for integrating them. > > > > > > == MUDROD == > > > '''Core''' > > > com.google.code.gson gson 2.5 compile > > > jar false > > > org.jdom jdom 2.0.2 compile > > > jar false > > > org.elasticsearch elasticsearch 5.2.0 compile > > > jar false > > > org.elasticsearch elasticsearch-spark-20_2.11 5.2.0 compile > > > jar false > > > joda-time joda-time 2.9.4 compile > > > jar false > > > com.carrotsearch hppc 0.7.1 compile > > > jar false > > > org.apache.spark spark-core_2.11 2.1.0 compile > > > jar false > > > org.apache.spark spark-sql_2.11 2.1.0 compile > > > jar false > > > org.apache.spark spark-mllib_2.11 2.1.0 compile > > > jar false > > > org.scala-lang scala-library 2.11.8 compile > > > jar false > > > org.codehaus.jettison jettison 1.3.8 compile > > > jar false > > > commons-cli commons-cli 1.2 compile > > > jar false > > > net.sf.opencsv opencsv 2.3 compile > > > jar false > > > org.apache.jena jena-core 3.3.0 compile > > > jar false > > > junit junit 4.12 test > > > jar false > > > > > > '''Service''' > > > gov.nasa.jpl.mudrod mudrod-core 0.0.1-SNAPSHOT compile > > > jar false > > > javax.servlet javax.servlet-api 3.1.0 provided > > > jar false > > > com.google.code.gson gson 2.5 compile > > > jar false > > > > > > '''Web''' > > > * AngularJS - MIT License > > > * BootstrapJS - MIT License > > > * jQueryJS - MIT License > > > * Underscore JS - MIT License > > > > > > == DOMS == > > > * Apache Solr version 5.5.1http://lucene.apache.org/solr/ > > > * EDGE https://github.com/dataplumber/edge > > > * NetCDF4 http://unidata.github.io/netcdf4-python/ > > > * Python 3.5 (NOTE: only partial support for py2.7) > > > > > > Non stdlib Python dependencies: > > > * Jinja2==2.9.5 > > > * python-dateutil==2.6.0 > > > * cython==0.25.2 > > > * numpy==1.12.0 > > > * scipy==0.18.1 > > > * netCDF4==1.2.7 > > > * solrpy3 > > > * siphon==0.4.0 > > > * neo4j-driver==1.1.0 > > > * matplotlib==2.0.0 > > > * requests==2.13.0 > > > * shapely==1.5.17 > > > * flask==0.12 > > > * networkx==1.11 > > > * pyproj==1.9.5.1 > > > * blist==1.3.6 > > > > > > == NEXUS == > > > '''Analysis''' > > > * https://github.com/dataplumber/nexus/blob/master/ > > > analysis/package-list.txt > > > * https://github.com/dataplumber/nexus/blob/master/ > > > analysis/requirements.txt > > > > > > '''Client''' > > > * https://github.com/dataplumber/nexus/blob/master/ > > > client/requirements.txt > > > > > > '''Climatology''' > > > * matplotlib > > > * numpy > > > * netCDF4 > > > * pathos (https://pypi.python.org/pypi/pathos) > > > > > > '''Data-access''' > > > * https://github.com/dataplumber/nexus/blob/master/ > > > data-access/requirements.txt > > > > > > '''Nexus-ingest''' > > > ''Dataset-tiler'' > > > * https://github.com/dataplumber/nexus/tree/master/ > > > nexus-ingest/dataset-tiler/build/reports > > > > > > ''developer-box'' > > > * Just a collection of scripts/vagrant file used to stand up a > developer > > > instance of nexus ingestion. No dependencies to report > > > > > > ''Groovy-scripts'' > > > * Collection of Groovy scripts that can be used as part of data > > > ingestion. They only rely on the standard Groovy library and the > > > ‘nexus-messages’ project > > > > > > ''Nexus-messages'' > > > * https://github.com/dataplumber/nexus/tree/master/ > > > nexus-ingest/nexus-messages/build/reports > > > > > > ''nexus-sink'' > > > * https://github.com/dataplumber/nexus/tree/master/ > > > nexus-ingest/nexus-sink/build/reports > > > > > > ''nexus-xd-python-modules'' > > > * https://github.com/dataplumber/nexus/blob/master/ > > > nexus-ingest/nexus-xd-python-modules/package-list.txt > > > * https://github.com/dataplumber/nexus/blob/master/ > > > nexus-ingest/nexus-xd-python-modules/requirements.txt > > > > > > ''spring-xd-python'' > > > * only python standard libraries are used > > > > > > ''tcp-shell'' > > > * https://github.com/dataplumber/nexus/tree/master/ > > > nexus-ingest/tcp-shell/build/reports > > > > > > '''tools/deletebyquery''' > > > * https://github.com/dataplumber/nexus/blob/master/ > tools/deletebyquery/ > > > requirements.txt > > > > > > = Required Resources = > > > Mailing Lists > > > * priv...@sdap.incubator.apache.org > > > * d...@sdap.incubator.apache.org > > > * comm...@sdap.incubator.apache.org > > > > > > Git Repos > > > * https://git-wip-us.apache.org/repos/asf/incubator-nexus.git > > > * https://git-wip-us.apache.org/repos/asf/incubator-doms.git > > > * https://git-wip-us.apache.org/repos/asf/incubator-mudrod.git > > > > > > Issue Tracking > > > * JIRA Science Data Analytics Platform (SDAP) > > > > > > Continuous Integration > > > * Jenkins builds on https://builds.apache.org/ > > > > > > Web > > > * http://sdap.incubator.apache.org/ > > > * wiki at http://cwiki.apache.org > > > > > > = Initial Committers = > > > The following is a list of the planned initial Apache committers (the > > > active subset of the committers for the current repository on Github). > > > * Lewis John McGibbney (lewi...@apache.org) > > > * Vardis M. Tsontos (vardis.m.tson...@jpl.nasa.gov) > > > * Joseph C. Jacob (joseph.c.ja...@jpl.nasa.gov) > > > * Ed Armstrong (edward.m.armstr...@jpl.nasa.gov) > > > * Frank Greguska (gregu...@jpl.nasa.gov) > > > * Brian Wilson (brian.wil...@jpl.nasa.gov) > > > * Chaowe Phil Yang (cya...@gmu.edu) > > > * Yongyao Jiang (yjia...@gmu.edu) > > > * Yun Li (yl...@gmu.edu) > > > * Shawn R. Smith (sm...@coaps.fsu.edu) > > > * Jocelyn Elya (je...@coaps.fsu.edu) > > > * Mark Bourassa (boura...@coaps.fsu.edu) > > > * Thomas Cram (tc...@ucar.edu) > > > * Thomas Huang (thomas.hu...@jpl.nasa.gov) > > > * Steven Worley (wor...@ucar.edu) > > > * Zaihua Ji (z...@ucar.edu) > > > > > > = Affiliations = > > > NASA JPL > > > * Lewis John McGibbney (lewi...@apache.org) > > > * Vardis M. Tsontos (vardis.m.tson...@jpl.nasa.gov) > > > * Joseph C. Jacob (joseph.c.ja...@jpl.nasa.gov) > > > * Ed Armstrong (edward.m.armstr...@jpl.nasa.gov) > > > * Frank Greguska (gregu...@jpl.nasa.gov) > > > * Thomas Huang (thomas.hu...@jpl.nasa.gov) > > > * Brian Wilson (brian.wil...@jpl.nasa.gov) > > > > > > George Mason University > > > * Chaowe Phil Yang (cya...@gmu.edu) > > > * Yongyao Jiang (yjia...@gmu.edu) > > > * Yun Li (yl...@gmu.edu) > > > > > > Center for Ocean-Atmospheric Prediction Studies, Florida State > University > > > * Shawn R. Smith (sm...@coaps.fsu.edu) > > > * Jocelyn Elya (je...@coaps.fsu.edu) > > > * Mark Bourassa (boura...@coaps.fsu.edu) > > > > > > Computational Information Systems Laboratory (CISL) / National Center > for > > > Atmospheric Research (NCAR) > > > * Thomas Cram (tc...@ucar.edu) > > > * Zaihua Ji (z...@ucar.edu) > > > * Steven Worley (wor...@ucar.edu) > > > > > > = Sponsors = > > > > > > = Champion = > > > * Lewis McGibbney (NASA/JPL) > > > > > > = Nominated Mentors = > > > * TBD > > > * TBD > > > * TBD > > > > > > = Sponsoring Entity = > > > The Apache Incubator > > > > > > > > > -- > > > http://home.apache.org/~lewismc/ > > > @hectorMcSpector > > > http://www.linkedin.com/in/lmcgibbney > > > > > > > > > > > -- > > http://home.apache.org/~lewismc/ > > @hectorMcSpector > > http://www.linkedin.com/in/lmcgibbney > > >
