Hi Steve, It was not so much the lack of committers as it was the current diversity. That is not a blocker for entry to Incubation.
I am willing to be one of the Mentors. Once there are at least two more we can push forward. Regards, Dave > On Aug 1, 2017, at 5:09 AM, Steve Lawrence <stephen.d.lawre...@gmail.com> > wrote: > > Discussions have died down, and I think the consensus from the responses > is that the issues are 1) the lack of committers and 2) the lack of a > champion and mentors. We hope to address #1 and grow the community as > part of incubation. Is anyone interested in being a champion or mentor > and help us with #2? > > Thanks, > - Steve > > On 07/26/2017 04:06 PM, Chris Mattmann wrote: >> This sounds like a very interesting project. >> >> I don’t have the time to mentor at the moment but I will keep a close eye on >> it. >> >> Cheers, >> Chris Mattmann >> >> >> >> >> On 7/25/17, 11:53 AM, "McHenry, Kenton Guadron" <mche...@illinois.edu> wrote: >> >> Hi Dave, >> >> The developers that were at NCSA have moved on to other organizations. >> While we still leverage Daffodil and are very much interested in seeing it >> move forward, development is currently done by the Tresys team. Agreed on >> the synergy with Tika. >> >> Kenton McHenry, Ph.D. >> Principal Research Scientist, Adjunct Assistant Professor of Computer >> Science >> Deputy Director of the Scientific Software & Applications Division >> National Center for Supercomputing Applications, University of Illinois >> at Urbana-Champaign >> >> On Jul 24, 2017, at 1:55 PM, Dave Fisher >> <dave2w...@comcast.net<mailto:dave2w...@comcast.net>> wrote: >> >> Hi Kenton, >> >> Is there any reason that you and others from the NCSA are not Initial >> Committers? That would make this proposal stronger. >> >> Regarding Apache Tika - it relies on other projects including Apache POI >> and Apache PDFBox. They are pragmatic about what is used. If Daffodil works >> to expand then I think that there would be good synergy between the >> projects. I know as a POI PMC member that the POI community has >> significantly benefited from the Tika community some of whom are from Mitre. >> >> To date Tika has not emphasized structured data, although they do extract >> content from Excel and OpenOffice. >> >> I am intrigued. >> >> Regards, >> Dave >> >> On Jul 24, 2017, at 10:55 AM, McHenry, Kenton Guadron >> <mche...@illinois.edu<mailto:mche...@illinois.edu>> wrote: >> >> Yes, DFDL and its open source implementation Daffodil are more about file >> formats and getting access to the entirety of a file's contents in a >> consistent way through machine readable specifications. The work has >> implications in the area of digital preservation allowing one to preserve >> these machine readable specifications rather than all the tools needed to >> open/save a file in order to work with it. Imagine someone developing >> graphics software to work with 3D models and not having to worry about the >> hundreds of formats out there for 3D meshes (whether there are tools for >> opening the files and whether they can get access to those tools, whether >> the spec is available and worrying about how complex that spec is to >> implement, etc.), and simply building their code around the contents (e.g. >> vertices, faces, etc.). One could come up with similar scenarios for other >> data types (documents, images, videos, audio, depth data, numeric data). >> Ideally tools built supporting DFDL, could someday, support any format for >> that type without the developer having to worry about the details of how >> that data is represented within a file. >> >> Kenton McHenry, Ph.D. >> Principal Research Scientist, Adjunct Assistant Professor of Computer >> Science >> Deputy Director of the Scientific Software & Applications Division >> National Center for Supercomputing Applications, University of Illinois >> at Urbana-Champaign >> >> On Jul 24, 2017, at 10:30 AM, Steve Lawrence >> <stephen.d.lawre...@gmail.com<mailto:stephen.d.lawre...@gmail.com><mailto:stephen.d.lawre...@gmail.com>> >> wrote: >> >> I'll preface this saying that I don't have a ton of experience with >> Apache Tika. But based on my understanding, Tika and Daffodil do have >> somewhat similar goals, but reach them in different ways. For example, >> Tika requires that one writes /code/ to perform data extraction, usually >> relying on existing Java libraries to extract the desired metadata. The >> downside to this is that code can be buggy, and libraries might not even >> exist for formats of interest (especially common with legacy and >> military data). >> >> Daffodil, on the other hand, does not require one to write any code. >> Instead, one writes a DFDL Schema (similar to XML Schema, with DFDL >> annotations) that fully describes the data, which Daffodil then uses to >> convert the data to XML/JSON for extraction. So adding support for a new >> format means writing a new schema rather than new code. And less code >> generally means less bugs. Also, for secure systems that require >> certification, generally speaking, it is easier to certify a schema as >> compared to code. >> >> We certainly don't believe that Daffodil could replace Tika, but it does >> have the potential to add new functionality to Tika for formats that do >> not have existing libraries. One of our goals is to look into >> integrating Daffodil support into tools like Tika. We'd love to hear >> from Tika devs if this is something they'd be interested in. >> >> I'll also add that whereas Tika tends to focus primarily on metadata, >> DFDL schemas usually describe an entire file format down to the byte, so >> one can extract more than just meta data, including text and binary >> data. Further differentiating, Daffodil has support for serializing data >> (called unparse) from the XML/JSON representation, allowing one to >> transform or filter data as well. We don't believe this feature is all >> that applicable to Tika, but may be useful to other technologies such as >> filtering or data fuzzing technologies. >> >> - Steve >> >> >> On 07/24/2017 10:59 AM, Mike Drob wrote: >> What is the relationship between Daffodil and something like Apache Tika's >> extraction engine? >> >> On Mon, Jul 24, 2017 at 9:53 AM, Steve Lawrence < >> >> stephen.d.lawre...@gmail.com<mailto:stephen.d.lawre...@gmail.com><mailto:stephen.d.lawre...@gmail.com>> >> wrote: >> >> Dear Apache Incubator Community, >> >> We would like to start a discussion around a proposal to bring Daffodil >> into the Apache Incubator. Daffodil is a implementation of the DFDL >> specification used to convert between fixed format data and XML/JSON. >> >> The draft proposal can be found in the wiki at the following URL: >> >> https://wiki.apache.org/incubator/DaffodilProposal >> >> We do not yet have a champion or mentors, but it was recommended that we >> create a proposal and send it to this list to potentially find those >> that might be interested. The text for the draft proposal is found >> below. We look forward to your input. >> >> Thanks, >> -Steve >> >> >> = Daffodil Proposal = >> >> == Abstract == >> >> Daffodil is an implementation of the Data Format Description Language >> (DFDL) used to convert between fixed format data and XML/JSON. >> >> == Proposal == >> >> The Data Format Description Language (DFDL) is a specification, >> developed by the Open Grid Forum, capable of describing many data >> formats, including both textual and binary, scientific and numeric, >> legacy and modern, commercial record-oriented, and many industry and >> military standards. It defines a language that is a subset of W3C XML >> schema to describe the logical format of the data, and annotations >> within the schema to describe the physical representation. >> >> Daffodil is an open source implementation of the DFDL specification that >> uses these DFDL schemas to parse fixed format data into an infoset, >> which is most commonly represented as either XML or JSON. This allows >> the use of well-established XML or JSON technologies and libraries to >> consume, inspect, and manipulate fixed format data in existing >> solutions. Daffodil is also capable of the reverse by serializing or >> "unparsing" an XML or JSON infoset back to the original data format. >> >> == Background == >> >> Many different software solutions need to consume and manage data, >> including data directed routing, databases, data analysis, data >> cleansing, data visualizing, and more. A key aspect of such solutions is >> the need to transform the data into an easily consumable format. >> Usually, this means that for each unique data format, one develops a >> tool that can read and extract the necessary information, often leading >> to ad-hoc and data-format-specific description systems. Such systems are >> often proprietary, not well tested, and incompatible, leading to vendor >> lock-in, flawed software, and increased training costs. DFDL is a new >> standard, with version 1.0 completed in October of 2016, that solves >> these problems by defining an open standard to describe many different >> data formats and how to parse and unparse between the data and XML/JSON. >> >> Two closed source implementations of DFDL currently exist. The first was >> created by IBM and is now part of their IBM® Integration Bus product. >> The second was created by the European Space Agency, called DFDL4S or >> "DFDL for Space" targeted at the challenges of their satellite data >> processing. >> >> Around 2005, Pacific Northwest National Lab created Defuddle, built as >> an open source implementation and proof of concept of the draft DFDL >> specification and a test bed to feed new concepts into specification >> development. Primary development of Defuddle was eventually taken over >> by the National Center for Supercomputing Applications (NCSA). However, >> due to evolution of the DFDL specification and architectural and >> performance issues with Defuddle, around 2009, NCSA restarted the >> project with the new name of Daffodil, with a goal of implementing the >> complete DFDL specification. Daffodil development continued at NCSA >> until around 2012, at which point development slowed due to budget >> limitations. Shortly thereafter, primary development was picked up by >> Tresys Technology where it continues today, with contributions from >> other entities such as the Navy Research Lab, the Air Force Research >> Lab, MITRE, and Booz Allen Hamilton. In February of 2015, Daffodil >> version 1.0.0 was released, including support for the DFDL features >> needed to parse many common file formats. Daffodil version 2.0.0 is >> expected to be released in August of 2017, which will include unparse >> support with one-to-one parsing feature parity. >> >> Entities including IBM, MITRE, NATO NCI Agency, Northrop-Grumman, Quark >> Security, Raytheon, and Tresys Technology have developed DFDL schemas >> for many data formats from varying technology domains, including PNG, >> GIF, BMP, PCAP, HL7, EDIFACT, NACHA, vCard, iCalendar, and MIL-STD-2045, >> many of which are publicly available on the DFDL Schemas github. There >> are also a number of military-application data formats, the >> specifications of which are not public, which have historically been >> very difficult and expensive to process, and for which DFDL schemas have >> been created or are actively in development; these include >> MIL-STD-6040/USMTF ATO, MIL-STD-6017/VMF, MIL-STD-6016/NATO STANAG 5516 >> (aka "Link16"). >> >> == Rationale == >> >> Numerous software solutions exist that consume, inspect, analyze, and >> transform data, many of which can be found in the Apache Software >> Foundation (ASF). In order for tools like these to consume new types of >> data, custom extensions are usually required, often with high >> development and testing costs. Daffodil fills a clear gap in many of >> these solutions, providing a simple and low cost way to transform data >> to XML or JSON, which many of these tools natively support already. With >> the upcoming 2.0.0 release, the Daffodil project will have achieved a >> level of functionality in both parse and unparse that, when integrated >> into existing solutions, could provide for a new method to quickly >> enable support for new data formats. >> >> == Initial Goals == >> >> * Relicense the existing code from the University of Illinois/NCSA Open >> Source License to the Apache License version 2.0, working with Apache >> Legal to ensure correctness, and with Daffodil contributors to get >> their permission. >> * Move the existing codebase, documentation, bugs, and mailing lists to >> the Apache hosted infrastructure >> * Establish a formal release process and schedule, allowing for >> dependable release cycles in a manner consistent with the Apache >> development process. >> * Build relationships with ASF projects to add Daffodil support where >> appropriate >> * Grow the community to establish a diversity of background and expertise. >> >> == Current Status == >> >> === Meritocracy === >> >> All initial committers are familiar with the principles of meritocracy. >> The Daffodil project has followed the model of meritocracy in the past, >> providing multiple outside entities commit access based on the quality >> of their contributions. In order to grow the Daffodil user base and >> development community, we are dedicated to continuing to operate >> Daffodil as a meritocracy. >> >> A key ingredient in a meritocracy of developers is open group code >> review. The Daffodil project has operated in this mode throughout its >> existence and this provides a forum to improve the code, verify code >> quality, and educate new developers on the code base. >> >> === Community === >> >> Daffodil has a small community of users and developers. Although primary >> Daffodil development is done by Tresys Technology, a handful of other >> contributions have come from other entities including the Navy Research >> Lab, the Air Force Research Lab, MITRE, and Booz Allen Hamilton. In >> addition to developers, multiple users of Daffodil have created DFDL >> schemas, including entities such as MITRE, IBM, Raytheon, Quark >> Security, and Tresys Technology. The DFDL Schemas github community has >> been created as a place for DFDL schemas to be published. The Daffodil >> project also makes use of mailing lists, !HipChat, and Confluence >> Questions to build a community of users and system for support. >> >> === Core Developers === >> >> The core developers of Daffodil are employed by Tresys Technology. We >> will work to grow the community among a more diverse set of developers >> and industries. >> >> === Alignment === >> >> Daffodil was created as an open source project with a philosophy >> consistent with The Apache Way. A strong belief in meritocracy, >> community involvement in decisions, openness, and ensuring a high level >> of quality in code, documentation, and testing are some of our shared >> core beliefs. >> >> Further, as mentioned in the Rationale section, Daffodil fills a gap >> that exists in many ASF projects, including !NiFi, Spark, Storm, Hadoop, >> Tika, and others. In order for tools like these to consume new types of >> data, custom extensions are usually required. Rather than create such >> extensions, Daffodil provides an easy and standards-compliant way to >> transform data to XML or JSON, which many of these tools already >> natively support. >> >> == Known Risks == >> >> === Orphaned Products === >> >> The current core developers are the leading contributors in the space of >> DFDL and wish to see it flourish. Though there is some risk that the >> initial committers all come from the same company, a goal of entering >> into incubation is to grow the development community to minimize the >> risk of reliance on a single company. >> >> === Inexperience with Open Source === >> >> The Daffodil project began as an open source project and has continued >> that model throughout development. This includes public bug tracking, >> git revision control, automated builds and tests, and a public wiki for >> documentation. >> >> Additionally, the current core developers and initial committers all >> work for a company that relies on, believes in, promotes, and has led or >> contributed to many open source software projects, including SELinux >> Userspace, OpenSCAP, CLIP, refpolicy, setools, RPM, and others. As such, >> there is low risk related to inexperience with open source software and >> processes. >> >> === Homogeneous Developers === >> >> The proposed initial committers come from a single entity, though we are >> committed to growing the Daffodil development community to include a >> broad group of additional committers from a wide array of industries. >> >> === Reliance on Salaried Developers === >> >> The proposed initial committers are paid by their employer to contribute >> to the Daffodil project. We expect that Daffodil development will >> continue with salaried developers, and are committed to growing the >> community to include non-salaried developers as well. >> >> === Relationship with other Apache Projects === >> >> As mentioned in the Alignment section, Daffodil fills a clear gap in >> numerous other ASF projects that consume and manage large amounts of data. >> >> As a specific example, Daffodil developers have created a Daffodil >> Apache !NiFi Processor, currently in use in data transfer solutions, >> which allows one to ingest non-native data into an Apache !NiFi pipeline >> as XML or JSON. This processor was well received by the Apache !NiFi >> developers, with positive comments about the concise API and how it >> could handle non-native data. Daffodil developers have also successfully >> prototyped integration with Apache Spark. We believe Daffodil could >> provide a strong benefit to many other ASF projects that handle fixed >> format data. We anticipate working closely with such ASF projects to >> include Daffodil where applicable to increase their ability to support >> new data formats with minimal effort. >> >> Daffodil also depends on existing ASF projects, including Apache Commons >> and Apache Xerces. >> >> === An Excessive Fascination with the Apache Brand === >> >> Although the Apache brand may certainly help to attract more >> contributors, publicity is not the reason for this proposal. We believe >> Daffodil could provide a great benefit to the ASF and the numerous data >> focused projects that comprise it, as described in the Rationale and >> Alignment sections. We hope to build a strong and vibrant community >> built around The Apache Way, and not dependent on a single company. >> >> === Documentation === >> >> Daffodil documentation can be found at: >> >> * >> https://opensource.ncsa.illinois.edu/confluence/ >> display/DFDL/Daffodil%3A+Open+Source+DFDL >> >> Information about DFDL can be found at: >> >> * https://www.ogf.org/ogf/doku.php/standards/dfdl/dfdl >> * >> https://www.ibm.com/support/knowledgecenter/en/SSMKHH_9.0. >> 0/com.ibm.etools.mft.doc/df20060_.htm >> >> Public examples of DFDL Schemas can be found at: >> >> * https://github.com/DFDLSchemas >> >> == Initial Source == >> >> The Daffodil git repo goes back to mid-2011 with approximately 20 >> different contributors and feedback from many users and developers. The >> core codebase is written in Scala and includes both a Scala and Java >> API, along with Javadocs and Scaladocs for API usage. The initial code >> will come from the git repository currently hosted by NCSA at the >> University of Illinois : >> >> https://opensource.ncsa.illinois.edu/bitbucket/ >> projects/DFDL/repos/daffodil/ >> >> == Source and Intellectual Property Submission == >> >> The complete Daffodil code is licensed under the University of >> Illinois/NCSA Open Source License. Much of the current codebase has been >> developed by Tresys Technology, who is open to relicensing the code to >> the Apache License version 2.0 and donate the source to the ASF. >> Contacts at NCSA are also open to relicensing their contributions to >> Apache v2. We plan to contact the other contributors and ask for >> permission to relicense and donate their contributed code. For those >> that decline or we cannot contact, their code will be removed or >> replaced. We will work closely with Apache Legal to ensure all issues >> related to relicensing are acceptable. >> >> == External Dependencies == >> >> We believe all current dependencies are compatible with the ASF >> guidelines. Our dependency licenses come from the following license >> styles: Apache v2, BSD, MIT, and ICU. The list of current Daffodil >> dependencies and their licenses are documented here: >> >> https://opensource.ncsa.illinois.edu/confluence/ >> display/DFDL/Dependencies+and+Licenses >> >> == Cryptography == >> >> None >> >> == Required Resources == >> >> === Mailing Lists === >> >> * comm...@daffodil.incubator.apache.org >> * d...@daffodil.incubator.apache.org >> * priv...@daffodil.incubator.apache.org >> * u...@daffodil.incubator.apache.org >> >> === Source Control === >> >> git://git.apache.org/incubator-daffodil.git >> >> === Issue Tracking === >> >> JIRA Daffodil (DFDL) >> >> === Initial Committers === >> >> * Beth Finnegan <efinnegan at tresys dot com> >> * Dave Thompson <dthompson at tresys dot com> >> * Josh Adams <jadams at tresys dot com> >> * Mike Beckerle <mbeckerle at tresys dot com> >> * Steve Lawrence <slawrence at tresys dot com> >> * Taylor Wise <twise at tresys dot com> >> >> === Affiliations === >> >> * Beth Finnegan (Tresys Technology) >> * Dave Thompson (Tresys Technology) >> * Josh Adams (Tresys Technology) >> * Mike Beckerle (Tresys Technology) >> * Steve Lawrence (Tresys Technology) >> * Taylor Wise (Tresys Technology) >> >> == Sponsors == >> >> === Champion === >> >> * TBD >> >> === Nominated Mentors === >> >> * TBD >> >> === Sponsoring Entity === >> >> We request the Apache Incubator to sponsor this project. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> For additional commands, e-mail: general-h...@incubator.apache.org >> >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: >> general-unsubscr...@incubator.apache.org<mailto:general-unsubscr...@incubator.apache.org> >> For additional commands, e-mail: >> general-h...@incubator.apache.org<mailto:general-h...@incubator.apache.org> >> >> >> >> >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> For additional commands, e-mail: general-h...@incubator.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org
signature.asc
Description: Message signed with OpenPGP
