On Tue, Jun 9, 2015 at 9:14 PM, Joe Witt <[email protected]> wrote: > Russell, > > This sounds like a great idea. I'm no clojure expert but it seems > like it would be quite reasonable to build processors in Clojure. > We've also discussed doing something similar in Scala. Basically > languages that run on the JVM we should have a good chance of > providing a nice developer experience for. >
Joe, it's great to hear that: not all ASF projects seem to be as warmly welcoming of the prospect of non-Java JVM-targeted language based contributions into their codebases, often for understandable reasons of lack of familiarity, the perceived complexity of having to integrate outside their standard toolchain (though actual Maven<->Leiningen integration is easier than most developers think), etc. > It is quite common for folks in the earlier stages of learning nifi to > utilize ExecuteStreamCommand, ExecuteProcess, etc.. type of > processors. This is a nice and gradual transition model. Which I'd seen mentioned over several weeks of lurking, and with which I'd been favorably impressed. I wish Hadoop had a similar type of gradual transition model; it would have made the "activation energy" barrier to developing for parts of it (e.g. Hive SimpleUDFs and GenericUDFs and Oozie custom ActionExecutors in my own case) more easily surmountable. > But we do > like to help avoid having to make those external calls over time. > Indeed! > Would you be interested in collaborating on putting together some nice > examples or talking about how we could support clojure nicely? > I'd love to help/collaborate in any way. I'm not sure about putting together some examples at the moment, since I've not written anything yet for Nifi; do you have anything in particular in mind which needs doing which might be a useful first test of the approach? Cheers, Russell > Thanks > Joe > > On Tue, Jun 9, 2015 at 3:27 PM, Russell Whitaker > <[email protected]> wrote: >> At work, I've found the sweet spot for Clojure programming in our Hadoop >> data processing stack: writing Hive UDFs (user-defined functions) which get >> distributed to HDFS, registered ("add jar ..."), and invoked as needed by >> users. >> It's been a real treat to avoid having to write Java (which I can do >> and have done >> much of in a past life) but still interoperate in the JVM. >> >> Now we're adding Nifi as a generalized data ingestion system into our Hadoop >> processing clusters, with various sources and (mostly) PutHDFS targets >> (hoping >> to do PutS3 in future), and are wondering how we might consider our team's >> emerging development pattern of Clojure coding to the JVM and plugging into >> an >> otherwise "pure Java" framework; i.e. we'd like to explore doing under >> Nifi what >> we've done under Hadoop. >> >> So far, the only option that really comes to mind is invoking "java >> -cp <my_clojure_lib>" >> in the context of an ExecuteStream processor, which is fine as things >> go - and how >> we're likely to quickly prototype new core libs for business-specific >> ingestion >> logic in Nifi - but I'm wondering if there's been some >> as-yet-undiscussed thinking on >> this matter in the Nifi community. >> >> Thanks, R >> >> -- >> Russell Whitaker >> http://twitter.com/OrthoNormalRuss >> http://github.com/russellwhitaker -- Russell Whitaker http://twitter.com/OrthoNormalRuss http://www.linkedin.com/pub/russell-whitaker/0/b86/329
