Thanks John! On 08/02/2017 03:23 PM, John D. Ament wrote: > You can also count me in as a mentor. > > John > > On Wed, Aug 2, 2017 at 3:14 PM Steve Lawrence <stephen.d.lawre...@gmail.com> > wrote: > >> Understood. Thanks for the interest! >> >> - Steve >> >> On 08/02/2017 02:57 PM, Dave Fisher wrote: >>> Hi Steve, >>> >>> It was not so much the lack of committers as it was the current >> diversity. That is not a blocker for entry to Incubation. >>> >>> I am willing to be one of the Mentors. Once there are at least two more >> we can push forward. >>> >>> Regards, >>> Dave >>> >>>> On Aug 1, 2017, at 5:09 AM, Steve Lawrence < >> stephen.d.lawre...@gmail.com> wrote: >>>> >>>> Discussions have died down, and I think the consensus from the responses >>>> is that the issues are 1) the lack of committers and 2) the lack of a >>>> champion and mentors. We hope to address #1 and grow the community as >>>> part of incubation. Is anyone interested in being a champion or mentor >>>> and help us with #2? >>>> >>>> Thanks, >>>> - Steve >>>> >>>> On 07/26/2017 04:06 PM, Chris Mattmann wrote: >>>>> This sounds like a very interesting project. >>>>> >>>>> I don’t have the time to mentor at the moment but I will keep a close >> eye on it. >>>>> >>>>> Cheers, >>>>> Chris Mattmann >>>>> >>>>> >>>>> >>>>> >>>>> On 7/25/17, 11:53 AM, "McHenry, Kenton Guadron" <mche...@illinois.edu> >> wrote: >>>>> >>>>> Hi Dave, >>>>> >>>>> The developers that were at NCSA have moved on to other >> organizations. While we still leverage Daffodil and are very much >> interested in seeing it move forward, development is currently done by the >> Tresys team. Agreed on the synergy with Tika. >>>>> >>>>> Kenton McHenry, Ph.D. >>>>> Principal Research Scientist, Adjunct Assistant Professor of >> Computer Science >>>>> Deputy Director of the Scientific Software & Applications Division >>>>> National Center for Supercomputing Applications, University of >> Illinois at Urbana-Champaign >>>>> >>>>> On Jul 24, 2017, at 1:55 PM, Dave Fisher <dave2w...@comcast.net >> <mailto:dave2w...@comcast.net>> wrote: >>>>> >>>>> Hi Kenton, >>>>> >>>>> Is there any reason that you and others from the NCSA are not >> Initial Committers? That would make this proposal stronger. >>>>> >>>>> Regarding Apache Tika - it relies on other projects including >> Apache POI and Apache PDFBox. They are pragmatic about what is used. If >> Daffodil works to expand then I think that there would be good synergy >> between the projects. I know as a POI PMC member that the POI community has >> significantly benefited from the Tika community some of whom are from Mitre. >>>>> >>>>> To date Tika has not emphasized structured data, although they do >> extract content from Excel and OpenOffice. >>>>> >>>>> I am intrigued. >>>>> >>>>> Regards, >>>>> Dave >>>>> >>>>> On Jul 24, 2017, at 10:55 AM, McHenry, Kenton Guadron < >> mche...@illinois.edu<mailto:mche...@illinois.edu>> wrote: >>>>> >>>>> Yes, DFDL and its open source implementation Daffodil are more >> about file formats and getting access to the entirety of a file's contents >> in a consistent way through machine readable specifications. The work has >> implications in the area of digital preservation allowing one to preserve >> these machine readable specifications rather than all the tools needed to >> open/save a file in order to work with it. Imagine someone developing >> graphics software to work with 3D models and not having to worry about the >> hundreds of formats out there for 3D meshes (whether there are tools for >> opening the files and whether they can get access to those tools, whether >> the spec is available and worrying about how complex that spec is to >> implement, etc.), and simply building their code around the contents (e.g. >> vertices, faces, etc.). One could come up with similar scenarios for other >> data types (documents, images, videos, audio, depth data, numeric data). >> Ideally tools built supporting DFDL, could someday, support any format for >> that type without the developer having to worry about the details of how >> that data is represented within a file. >>>>> >>>>> Kenton McHenry, Ph.D. >>>>> Principal Research Scientist, Adjunct Assistant Professor of >> Computer Science >>>>> Deputy Director of the Scientific Software & Applications Division >>>>> National Center for Supercomputing Applications, University of >> Illinois at Urbana-Champaign >>>>> >>>>> On Jul 24, 2017, at 10:30 AM, Steve Lawrence < >> stephen.d.lawre...@gmail.com<mailto:stephen.d.lawre...@gmail.com><mailto: >> stephen.d.lawre...@gmail.com>> wrote: >>>>> >>>>> I'll preface this saying that I don't have a ton of experience with >>>>> Apache Tika. But based on my understanding, Tika and Daffodil do >> have >>>>> somewhat similar goals, but reach them in different ways. For >> example, >>>>> Tika requires that one writes /code/ to perform data extraction, >> usually >>>>> relying on existing Java libraries to extract the desired metadata. >> The >>>>> downside to this is that code can be buggy, and libraries might not >> even >>>>> exist for formats of interest (especially common with legacy and >>>>> military data). >>>>> >>>>> Daffodil, on the other hand, does not require one to write any code. >>>>> Instead, one writes a DFDL Schema (similar to XML Schema, with DFDL >>>>> annotations) that fully describes the data, which Daffodil then >> uses to >>>>> convert the data to XML/JSON for extraction. So adding support for >> a new >>>>> format means writing a new schema rather than new code. And less >> code >>>>> generally means less bugs. Also, for secure systems that require >>>>> certification, generally speaking, it is easier to certify a schema >> as >>>>> compared to code. >>>>> >>>>> We certainly don't believe that Daffodil could replace Tika, but it >> does >>>>> have the potential to add new functionality to Tika for formats >> that do >>>>> not have existing libraries. One of our goals is to look into >>>>> integrating Daffodil support into tools like Tika. We'd love to hear >>>>> from Tika devs if this is something they'd be interested in. >>>>> >>>>> I'll also add that whereas Tika tends to focus primarily on >> metadata, >>>>> DFDL schemas usually describe an entire file format down to the >> byte, so >>>>> one can extract more than just meta data, including text and binary >>>>> data. Further differentiating, Daffodil has support for serializing >> data >>>>> (called unparse) from the XML/JSON representation, allowing one to >>>>> transform or filter data as well. We don't believe this feature is >> all >>>>> that applicable to Tika, but may be useful to other technologies >> such as >>>>> filtering or data fuzzing technologies. >>>>> >>>>> - Steve >>>>> >>>>> >>>>> On 07/24/2017 10:59 AM, Mike Drob wrote: >>>>> What is the relationship between Daffodil and something like Apache >> Tika's >>>>> extraction engine? >>>>> >>>>> On Mon, Jul 24, 2017 at 9:53 AM, Steve Lawrence < >>>>> stephen.d.lawre...@gmail.com<mailto:stephen.d.lawre...@gmail.com >>> <mailto:stephen.d.lawre...@gmail.com>> wrote: >>>>> >>>>> Dear Apache Incubator Community, >>>>> >>>>> We would like to start a discussion around a proposal to bring >> Daffodil >>>>> into the Apache Incubator. Daffodil is a implementation of the DFDL >>>>> specification used to convert between fixed format data and >> XML/JSON. >>>>> >>>>> The draft proposal can be found in the wiki at the following URL: >>>>> >>>>> https://wiki.apache.org/incubator/DaffodilProposal >>>>> >>>>> We do not yet have a champion or mentors, but it was recommended >> that we >>>>> create a proposal and send it to this list to potentially find those >>>>> that might be interested. The text for the draft proposal is found >>>>> below. We look forward to your input. >>>>> >>>>> Thanks, >>>>> -Steve >>>>> >>>>> >>>>> = Daffodil Proposal = >>>>> >>>>> == Abstract == >>>>> >>>>> Daffodil is an implementation of the Data Format Description >> Language >>>>> (DFDL) used to convert between fixed format data and XML/JSON. >>>>> >>>>> == Proposal == >>>>> >>>>> The Data Format Description Language (DFDL) is a specification, >>>>> developed by the Open Grid Forum, capable of describing many data >>>>> formats, including both textual and binary, scientific and numeric, >>>>> legacy and modern, commercial record-oriented, and many industry and >>>>> military standards. It defines a language that is a subset of W3C >> XML >>>>> schema to describe the logical format of the data, and annotations >>>>> within the schema to describe the physical representation. >>>>> >>>>> Daffodil is an open source implementation of the DFDL specification >> that >>>>> uses these DFDL schemas to parse fixed format data into an infoset, >>>>> which is most commonly represented as either XML or JSON. This >> allows >>>>> the use of well-established XML or JSON technologies and libraries >> to >>>>> consume, inspect, and manipulate fixed format data in existing >>>>> solutions. Daffodil is also capable of the reverse by serializing or >>>>> "unparsing" an XML or JSON infoset back to the original data format. >>>>> >>>>> == Background == >>>>> >>>>> Many different software solutions need to consume and manage data, >>>>> including data directed routing, databases, data analysis, data >>>>> cleansing, data visualizing, and more. A key aspect of such >> solutions is >>>>> the need to transform the data into an easily consumable format. >>>>> Usually, this means that for each unique data format, one develops a >>>>> tool that can read and extract the necessary information, often >> leading >>>>> to ad-hoc and data-format-specific description systems. Such >> systems are >>>>> often proprietary, not well tested, and incompatible, leading to >> vendor >>>>> lock-in, flawed software, and increased training costs. DFDL is a >> new >>>>> standard, with version 1.0 completed in October of 2016, that solves >>>>> these problems by defining an open standard to describe many >> different >>>>> data formats and how to parse and unparse between the data and >> XML/JSON. >>>>> >>>>> Two closed source implementations of DFDL currently exist. The >> first was >>>>> created by IBM and is now part of their IBM® Integration Bus >> product. >>>>> The second was created by the European Space Agency, called DFDL4S >> or >>>>> "DFDL for Space" targeted at the challenges of their satellite data >>>>> processing. >>>>> >>>>> Around 2005, Pacific Northwest National Lab created Defuddle, built >> as >>>>> an open source implementation and proof of concept of the draft DFDL >>>>> specification and a test bed to feed new concepts into specification >>>>> development. Primary development of Defuddle was eventually taken >> over >>>>> by the National Center for Supercomputing Applications (NCSA). >> However, >>>>> due to evolution of the DFDL specification and architectural and >>>>> performance issues with Defuddle, around 2009, NCSA restarted the >>>>> project with the new name of Daffodil, with a goal of implementing >> the >>>>> complete DFDL specification. Daffodil development continued at NCSA >>>>> until around 2012, at which point development slowed due to budget >>>>> limitations. Shortly thereafter, primary development was picked up >> by >>>>> Tresys Technology where it continues today, with contributions from >>>>> other entities such as the Navy Research Lab, the Air Force Research >>>>> Lab, MITRE, and Booz Allen Hamilton. In February of 2015, Daffodil >>>>> version 1.0.0 was released, including support for the DFDL features >>>>> needed to parse many common file formats. Daffodil version 2.0.0 is >>>>> expected to be released in August of 2017, which will include >> unparse >>>>> support with one-to-one parsing feature parity. >>>>> >>>>> Entities including IBM, MITRE, NATO NCI Agency, Northrop-Grumman, >> Quark >>>>> Security, Raytheon, and Tresys Technology have developed DFDL >> schemas >>>>> for many data formats from varying technology domains, including >> PNG, >>>>> GIF, BMP, PCAP, HL7, EDIFACT, NACHA, vCard, iCalendar, and >> MIL-STD-2045, >>>>> many of which are publicly available on the DFDL Schemas github. >> There >>>>> are also a number of military-application data formats, the >>>>> specifications of which are not public, which have historically been >>>>> very difficult and expensive to process, and for which DFDL schemas >> have >>>>> been created or are actively in development; these include >>>>> MIL-STD-6040/USMTF ATO, MIL-STD-6017/VMF, MIL-STD-6016/NATO STANAG >> 5516 >>>>> (aka "Link16"). >>>>> >>>>> == Rationale == >>>>> >>>>> Numerous software solutions exist that consume, inspect, analyze, >> and >>>>> transform data, many of which can be found in the Apache Software >>>>> Foundation (ASF). In order for tools like these to consume new >> types of >>>>> data, custom extensions are usually required, often with high >>>>> development and testing costs. Daffodil fills a clear gap in many of >>>>> these solutions, providing a simple and low cost way to transform >> data >>>>> to XML or JSON, which many of these tools natively support already. >> With >>>>> the upcoming 2.0.0 release, the Daffodil project will have achieved >> a >>>>> level of functionality in both parse and unparse that, when >> integrated >>>>> into existing solutions, could provide for a new method to quickly >>>>> enable support for new data formats. >>>>> >>>>> == Initial Goals == >>>>> >>>>> * Relicense the existing code from the University of Illinois/NCSA >> Open >>>>> Source License to the Apache License version 2.0, working with >> Apache >>>>> Legal to ensure correctness, and with Daffodil contributors to get >>>>> their permission. >>>>> * Move the existing codebase, documentation, bugs, and mailing >> lists to >>>>> the Apache hosted infrastructure >>>>> * Establish a formal release process and schedule, allowing for >>>>> dependable release cycles in a manner consistent with the Apache >>>>> development process. >>>>> * Build relationships with ASF projects to add Daffodil support >> where >>>>> appropriate >>>>> * Grow the community to establish a diversity of background and >> expertise. >>>>> >>>>> == Current Status == >>>>> >>>>> === Meritocracy === >>>>> >>>>> All initial committers are familiar with the principles of >> meritocracy. >>>>> The Daffodil project has followed the model of meritocracy in the >> past, >>>>> providing multiple outside entities commit access based on the >> quality >>>>> of their contributions. In order to grow the Daffodil user base and >>>>> development community, we are dedicated to continuing to operate >>>>> Daffodil as a meritocracy. >>>>> >>>>> A key ingredient in a meritocracy of developers is open group code >>>>> review. The Daffodil project has operated in this mode throughout >> its >>>>> existence and this provides a forum to improve the code, verify code >>>>> quality, and educate new developers on the code base. >>>>> >>>>> === Community === >>>>> >>>>> Daffodil has a small community of users and developers. Although >> primary >>>>> Daffodil development is done by Tresys Technology, a handful of >> other >>>>> contributions have come from other entities including the Navy >> Research >>>>> Lab, the Air Force Research Lab, MITRE, and Booz Allen Hamilton. In >>>>> addition to developers, multiple users of Daffodil have created DFDL >>>>> schemas, including entities such as MITRE, IBM, Raytheon, Quark >>>>> Security, and Tresys Technology. The DFDL Schemas github community >> has >>>>> been created as a place for DFDL schemas to be published. The >> Daffodil >>>>> project also makes use of mailing lists, !HipChat, and Confluence >>>>> Questions to build a community of users and system for support. >>>>> >>>>> === Core Developers === >>>>> >>>>> The core developers of Daffodil are employed by Tresys Technology. >> We >>>>> will work to grow the community among a more diverse set of >> developers >>>>> and industries. >>>>> >>>>> === Alignment === >>>>> >>>>> Daffodil was created as an open source project with a philosophy >>>>> consistent with The Apache Way. A strong belief in meritocracy, >>>>> community involvement in decisions, openness, and ensuring a high >> level >>>>> of quality in code, documentation, and testing are some of our >> shared >>>>> core beliefs. >>>>> >>>>> Further, as mentioned in the Rationale section, Daffodil fills a gap >>>>> that exists in many ASF projects, including !NiFi, Spark, Storm, >> Hadoop, >>>>> Tika, and others. In order for tools like these to consume new >> types of >>>>> data, custom extensions are usually required. Rather than create >> such >>>>> extensions, Daffodil provides an easy and standards-compliant way to >>>>> transform data to XML or JSON, which many of these tools already >>>>> natively support. >>>>> >>>>> == Known Risks == >>>>> >>>>> === Orphaned Products === >>>>> >>>>> The current core developers are the leading contributors in the >> space of >>>>> DFDL and wish to see it flourish. Though there is some risk that the >>>>> initial committers all come from the same company, a goal of >> entering >>>>> into incubation is to grow the development community to minimize the >>>>> risk of reliance on a single company. >>>>> >>>>> === Inexperience with Open Source === >>>>> >>>>> The Daffodil project began as an open source project and has >> continued >>>>> that model throughout development. This includes public bug >> tracking, >>>>> git revision control, automated builds and tests, and a public wiki >> for >>>>> documentation. >>>>> >>>>> Additionally, the current core developers and initial committers all >>>>> work for a company that relies on, believes in, promotes, and has >> led or >>>>> contributed to many open source software projects, including SELinux >>>>> Userspace, OpenSCAP, CLIP, refpolicy, setools, RPM, and others. As >> such, >>>>> there is low risk related to inexperience with open source software >> and >>>>> processes. >>>>> >>>>> === Homogeneous Developers === >>>>> >>>>> The proposed initial committers come from a single entity, though >> we are >>>>> committed to growing the Daffodil development community to include a >>>>> broad group of additional committers from a wide array of >> industries. >>>>> >>>>> === Reliance on Salaried Developers === >>>>> >>>>> The proposed initial committers are paid by their employer to >> contribute >>>>> to the Daffodil project. We expect that Daffodil development will >>>>> continue with salaried developers, and are committed to growing the >>>>> community to include non-salaried developers as well. >>>>> >>>>> === Relationship with other Apache Projects === >>>>> >>>>> As mentioned in the Alignment section, Daffodil fills a clear gap in >>>>> numerous other ASF projects that consume and manage large amounts >> of data. >>>>> >>>>> As a specific example, Daffodil developers have created a Daffodil >>>>> Apache !NiFi Processor, currently in use in data transfer solutions, >>>>> which allows one to ingest non-native data into an Apache !NiFi >> pipeline >>>>> as XML or JSON. This processor was well received by the Apache !NiFi >>>>> developers, with positive comments about the concise API and how it >>>>> could handle non-native data. Daffodil developers have also >> successfully >>>>> prototyped integration with Apache Spark. We believe Daffodil could >>>>> provide a strong benefit to many other ASF projects that handle >> fixed >>>>> format data. We anticipate working closely with such ASF projects to >>>>> include Daffodil where applicable to increase their ability to >> support >>>>> new data formats with minimal effort. >>>>> >>>>> Daffodil also depends on existing ASF projects, including Apache >> Commons >>>>> and Apache Xerces. >>>>> >>>>> === An Excessive Fascination with the Apache Brand === >>>>> >>>>> Although the Apache brand may certainly help to attract more >>>>> contributors, publicity is not the reason for this proposal. We >> believe >>>>> Daffodil could provide a great benefit to the ASF and the numerous >> data >>>>> focused projects that comprise it, as described in the Rationale and >>>>> Alignment sections. We hope to build a strong and vibrant community >>>>> built around The Apache Way, and not dependent on a single company. >>>>> >>>>> === Documentation === >>>>> >>>>> Daffodil documentation can be found at: >>>>> >>>>> * >>>>> https://opensource.ncsa.illinois.edu/confluence/ >>>>> display/DFDL/Daffodil%3A+Open+Source+DFDL >>>>> >>>>> Information about DFDL can be found at: >>>>> >>>>> * https://www.ogf.org/ogf/doku.php/standards/dfdl/dfdl >>>>> * >>>>> https://www.ibm.com/support/knowledgecenter/en/SSMKHH_9.0. >>>>> 0/com.ibm.etools.mft.doc/df20060_.htm >>>>> >>>>> Public examples of DFDL Schemas can be found at: >>>>> >>>>> * https://github.com/DFDLSchemas >>>>> >>>>> == Initial Source == >>>>> >>>>> The Daffodil git repo goes back to mid-2011 with approximately 20 >>>>> different contributors and feedback from many users and developers. >> The >>>>> core codebase is written in Scala and includes both a Scala and Java >>>>> API, along with Javadocs and Scaladocs for API usage. The initial >> code >>>>> will come from the git repository currently hosted by NCSA at the >>>>> University of Illinois : >>>>> >>>>> https://opensource.ncsa.illinois.edu/bitbucket/ >>>>> projects/DFDL/repos/daffodil/ >>>>> >>>>> == Source and Intellectual Property Submission == >>>>> >>>>> The complete Daffodil code is licensed under the University of >>>>> Illinois/NCSA Open Source License. Much of the current codebase has >> been >>>>> developed by Tresys Technology, who is open to relicensing the code >> to >>>>> the Apache License version 2.0 and donate the source to the ASF. >>>>> Contacts at NCSA are also open to relicensing their contributions to >>>>> Apache v2. We plan to contact the other contributors and ask for >>>>> permission to relicense and donate their contributed code. For those >>>>> that decline or we cannot contact, their code will be removed or >>>>> replaced. We will work closely with Apache Legal to ensure all >> issues >>>>> related to relicensing are acceptable. >>>>> >>>>> == External Dependencies == >>>>> >>>>> We believe all current dependencies are compatible with the ASF >>>>> guidelines. Our dependency licenses come from the following license >>>>> styles: Apache v2, BSD, MIT, and ICU. The list of current Daffodil >>>>> dependencies and their licenses are documented here: >>>>> >>>>> https://opensource.ncsa.illinois.edu/confluence/ >>>>> display/DFDL/Dependencies+and+Licenses >>>>> >>>>> == Cryptography == >>>>> >>>>> None >>>>> >>>>> == Required Resources == >>>>> >>>>> === Mailing Lists === >>>>> >>>>> * comm...@daffodil.incubator.apache.org >>>>> * d...@daffodil.incubator.apache.org >>>>> * priv...@daffodil.incubator.apache.org >>>>> * u...@daffodil.incubator.apache.org >>>>> >>>>> === Source Control === >>>>> >>>>> git://git.apache.org/incubator-daffodil.git >>>>> >>>>> === Issue Tracking === >>>>> >>>>> JIRA Daffodil (DFDL) >>>>> >>>>> === Initial Committers === >>>>> >>>>> * Beth Finnegan <efinnegan at tresys dot com> >>>>> * Dave Thompson <dthompson at tresys dot com> >>>>> * Josh Adams <jadams at tresys dot com> >>>>> * Mike Beckerle <mbeckerle at tresys dot com> >>>>> * Steve Lawrence <slawrence at tresys dot com> >>>>> * Taylor Wise <twise at tresys dot com> >>>>> >>>>> === Affiliations === >>>>> >>>>> * Beth Finnegan (Tresys Technology) >>>>> * Dave Thompson (Tresys Technology) >>>>> * Josh Adams (Tresys Technology) >>>>> * Mike Beckerle (Tresys Technology) >>>>> * Steve Lawrence (Tresys Technology) >>>>> * Taylor Wise (Tresys Technology) >>>>> >>>>> == Sponsors == >>>>> >>>>> === Champion === >>>>> >>>>> * TBD >>>>> >>>>> === Nominated Mentors === >>>>> >>>>> * TBD >>>>> >>>>> === Sponsoring Entity === >>>>> >>>>> We request the Apache Incubator to sponsor this project. >>>>> >>>>> >> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >>>>> For additional commands, e-mail: general-h...@incubator.apache.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> <mailto:general-unsubscr...@incubator.apache.org> >>>>> For additional commands, e-mail: general-h...@incubator.apache.org >> <mailto:general-h...@incubator.apache.org> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >>>>> For additional commands, e-mail: general-h...@incubator.apache.org >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >>>> For additional commands, e-mail: general-h...@incubator.apache.org >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org >> For additional commands, e-mail: general-h...@incubator.apache.org >> >
--------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
