Ashish, Yes that sounds good and thanks for your interest in contributing. Please feel free to ask questions early and often.
Thanks and look forward to working with you Joe On Thu, Feb 19, 2015 at 10:00 PM, Ashish <[email protected]> wrote: > Joe, > > other way round is to support it, but let the End User download and > drop the binaries needed (not shipping them by default). > > I am keen on working to get the JMX support and based on my reading of > the code so far, it would be a Reporting Task (shall keep it generic > around Metrics) and add initially enable it only for JMX and later as > we have more clarity, we can add more functions. > > Shall I create a JIRA for this? > > On Fri, Feb 20, 2015 at 3:15 AM, Joe Witt <[email protected]> wrote: >> yeah absolutely - i was just addressing the fact that there is a >> licensing issue (in general) with that library. >> >> We can still totally do fun things with JMX, ganglia, etc.. Just >> takes a little more work when the best libs aren't' readily useable. >> >> Thanks >> Joe >> >> On Thu, Feb 19, 2015 at 4:42 PM, Andrew Purtell <[email protected]> wrote: >>> Would it suffice to not support Ganglia reporting? Then you'd not be >>> bringing in metrics-ganglia* and its transitive dependencies? >>> >>> * - And can actively exclude it in POMs if necessary >>> >>> >>> >>> On Thu, Feb 19, 2015 at 1:39 PM, Joe Witt <[email protected]> wrote: >>> >>>> Otis, >>>> >>>> We'll take a fresh look at JMX but if you want to help contribute in >>>> this area that would be awesome as well. >>>> >>>> Regarding the Metric library and its licensing fitness: >>>> ---------------------------- >>>> >>>> It's important to keep in mind that transitive dependencies and their >>>> licenses are just as important as direct dependencies. If you start >>>> here: >>>> >>>> https://github.com/dropwizard/metrics [yep good this one is ASLv2] >>>> from that we want to use >>>> >>>> https://github.com/dropwizard/metrics/tree/master/metrics-ganglia [yep >>>> good still ASLv2 from parent but let's look at deps] >>>> specifically this one: info.ganglia.gmetric4j:gmetric4j >>>> >>>> https://github.com/ganglia/gmetric4j [great it is BSD licensed and >>>> thats ok too as it is Category-A but lets check its deps] >>>> specifically this one: org.acplt.remotetea:remotetea-oncrpc >>>> >>>> http://sourceforge.net/projects/remotetea/ [And this is where things >>>> are problematic. This one is LGPLv2 which is Category-X - not allowed >>>> for either source or binary dependency] >>>> Upon further review the source does indeed depend on the remotetea >>>> library so we can't simply ignore that dependency. Which means we can >>>> use the gmetric4j library and thus can't use metrics-ganglia library. >>>> >>>> Now this is just from the ganglia tree. Other parts of the metrics >>>> library might well be fine. We're getting around this problem today >>>> by using the older metrics library and its older ganglia support which >>>> doesn't have this dependency licensing issue. >>>> >>>> >>>> Thanks >>>> Joe >>>> >>>> On Thu, Feb 19, 2015 at 3:56 PM, Otis Gospodnetic >>>> <[email protected]> wrote: >>>> > +1 to everything that Andrew said. >>>> > I think there is actually no licensing issue with Metric library - it's >>>> > ASLv2 - https://github.com/dropwizard/metrics/blob/master/LICENSE >>>> > >>>> > re JMX and auth stuff, we've documented it for SPM users here: >>>> > >>>> https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+Monitor+-+Standalone >>>> > >>>> > Otis >>>> > -- >>>> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management >>>> > Solr & Elasticsearch Support * http://sematext.com/ >>>> > >>>> > >>>> > On Thu, Feb 19, 2015 at 1:12 PM, Andrew Purtell <[email protected]> >>>> wrote: >>>> > >>>> >> > That should certainly be doable. It just never seemed clear that JMX >>>> >> > was the way to go. I think the theory had always been that if we >>>> >> > expose these sorts of things through the REST API over which we have >>>> >> > clearly defined authentication/authorization schemes we could then >>>> >> > support a variety of clients (people, systems). >>>> >> >>>> >> That's definitely true. >>>> >> >>>> >> What we do over on HBase is wire up our metrics to JMX and provide a >>>> REST >>>> >> endpoint in every process that presents all mbean data as JSON. We don't >>>> >> set up JMX remote access, but let users edit configs for launch scripts >>>> if >>>> >> RMI JMX access is desired. This has some minor advantages in my opinion: >>>> >> - We don't need to do anything special to expose JVM runtime data like >>>> >> detailed information on GC pools and times. >>>> >> - If it is desired for some reason, devs/admins then do have the option >>>> to >>>> >> set up the usual properties in the environment for RMI to JMX using the >>>> >> standard options for auth and such. >>>> >> >>>> >> In my experience some shops do want to work with JMX, usually when other >>>> >> systems that export operational data via JMX are already under >>>> management. >>>> >> >>>> >> >>>> >> >>>> >> On Thu, Feb 19, 2015 at 9:52 AM, Joe Witt <[email protected]> wrote: >>>> >> >>>> >> > Andrew, >>>> >> > >>>> >> > That should certainly be doable. It just never seemed clear that JMX >>>> >> > was the way to go. I think the theory had always been that if we >>>> >> > expose these sorts of things through the REST API over which we have >>>> >> > clearly defined authentication/authorization schemes we could then >>>> >> > support a variety of clients (people, systems). >>>> >> > >>>> >> > I'll have to read up more on the options for security around JMX and >>>> >> > the sort of client interactions that opens up. That said, it would >>>> >> > also be good to have a sense of tradeoffs of opening up further >>>> >> > avenues of client interaction beyond the REST API. Once we get our >>>> >> > REST API docs legit again (had to pull due to licensing concerns) then >>>> >> > that should help a lot too. >>>> >> > >>>> >> > Thanks >>>> >> > Joe >>>> >> > >>>> >> > >>>> >> > On Thu, Feb 19, 2015 at 12:42 PM, Andrew Purtell <[email protected] >>>> > >>>> >> > wrote: >>>> >> > > Would it be possible to instantiate a Metrics JmxReporter for each >>>> >> > process? >>>> >> > > Sounds like enabling metrics export via JMX could be an interesting >>>> >> JIRA >>>> >> > at >>>> >> > > any rate. >>>> >> > > >>>> >> > > I did a quick search of the User, Developer, and Admin guides and >>>> there >>>> >> > are >>>> >> > > no occurrences of the string "JMX" >>>> >> > > >>>> >> > > On Tue, Feb 17, 2015 at 10:25 PM, Otis Gospodnetic < >>>> >> > > [email protected]> wrote: >>>> >> > > >>>> >> > >> Hi Joe, >>>> >> > >> >>>> >> > >> Kafka devs describe their MBeans here - >>>> >> > >> https://kafka.apache.org/08/ops.html >>>> >> > >> - section 6.5, which is good for people who want to understand what >>>> >> each >>>> >> > >> metric means. >>>> >> > >> It sounds like the reporting task stuff is all about the push. I >>>> >> think >>>> >> > >> pull may be better for us, so if you can document the end point (I >>>> >> > assume >>>> >> > >> we are talking about an HTTP/REST API? That returns various >>>> metrics >>>> >> in >>>> >> > >> JSON?), that would be very very helpful (and not just to us). >>>> >> > >> >>>> >> > >> Thanks, >>>> >> > >> Otis >>>> >> > >> -- >>>> >> > >> Monitoring * Alerting * Anomaly Detection * Centralized Log >>>> Management >>>> >> > >> Solr & Elasticsearch Support * http://sematext.com/ >>>> >> > >> >>>> >> > >> >>>> >> > >> On Wed, Feb 18, 2015 at 12:50 AM, Joe Witt <[email protected]> >>>> >> wrote: >>>> >> > >> >>>> >> > >> > Otis, >>>> >> > >> > >>>> >> > >> > We're quite light on documentation thus far for the Reporting >>>> Task >>>> >> > >> > API. But you can see a bit about it here: >>>> >> > >> > >>>> >> > >> > >>>> >> > >> > >>>> >> > >> >>>> >> > >>>> >> >>>> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks >>>> >> > >> > >>>> >> > >> > >>>> >> > >> >>>> >> > >>>> >> >>>> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task >>>> >> > >> > >>>> >> > >> > The javadocs are published to Maven as well so that will do >>>> nicely >>>> >> to >>>> >> > >> > give a sense of what we expose from the Internals of NiFi to an >>>> >> > >> > implementation of a reporting task. >>>> >> > >> > >>>> >> > >> > But feel free to ask detailed questions and as many as you want. >>>> >> That >>>> >> > >> > will give us good information on the types of things we need to >>>> >> cover >>>> >> > >> > and in how much detail. >>>> >> > >> > >>>> >> > >> > Thanks >>>> >> > >> > Joe >>>> >> > >> > >>>> >> > >> > On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic >>>> >> > >> > <[email protected]> wrote: >>>> >> > >> > > Hi Joe, >>>> >> > >> > > >>>> >> > >> > > On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[email protected] >>>> > >>>> >> > wrote: >>>> >> > >> > > >>>> >> > >> > >> Otis, >>>> >> > >> > >> >>>> >> > >> > >> Glad you are liking it. >>>> >> > >> > >> >>>> >> > >> > >> We do provide support for both push and pull based metrics >>>> >> > >> > >> reporting/gathering. >>>> >> > >> > >> >>>> >> > >> > >> For NiFi being able to push out metrics: >>>> >> > >> > >> We do have an API for what we call "Reporting Tasks". For >>>> >> example >>>> >> > we >>>> >> > >> > >> have a Ganglia reporter today. >>>> >> > >> > >> >>>> >> > >> > > >>>> >> > >> > > So we are talking about the Yammer/Coda Hale's Metrics being >>>> used >>>> >> > here? >>>> >> > >> > > If so, are the metrics also available via JMX, too? >>>> >> > >> > > >>>> >> > >> > > For a client being able to pull metrics from NiFI: >>>> >> > >> > >> The REST API is the way to go. >>>> >> > >> > >> >>>> >> > >> > > >>>> >> > >> > > This sounds good. Are this API and the metrics it exposes >>>> >> > documented >>>> >> > >> > > anywhere? >>>> >> > >> > > I looked in the guides, but couldn't find anything. >>>> >> > >> > > >>>> >> > >> > > >>>> >> > >> > >> Do either of these sound like a path that can work for you? >>>> Let >>>> >> us >>>> >> > >> > >> know if you'd like more details. >>>> >> > >> > >> >>>> >> > >> > > >>>> >> > >> > > Thanks, >>>> >> > >> > > Otis >>>> >> > >> > > -- >>>> >> > >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log >>>> >> > Management >>>> >> > >> > > Solr & Elasticsearch Support * http://sematext.com/ >>>> >> > >> > > >>>> >> > >> > > >>>> >> > >> > > >>>> >> > >> > > >>>> >> > >> > >> >>>> >> > >> > >> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic >>>> >> > >> > >> <[email protected]> wrote: >>>> >> > >> > >> > Hi, >>>> >> > >> > >> > >>>> >> > >> > >> > Nifi looks mightily impressive! >>>> >> > >> > >> > I see Nifi has built-in monitoring and metrics.... but is >>>> there >>>> >> > an >>>> >> > >> API >>>> >> > >> > >> that >>>> >> > >> > >> > would allow something like SPM to add support for Nifi >>>> >> > monitoring? >>>> >> > >> Or >>>> >> > >> > >> JMX? >>>> >> > >> > >> > >>>> >> > >> > >> > Thanks, >>>> >> > >> > >> > Otis >>>> >> > >> > >> > -- >>>> >> > >> > >> > Monitoring * Alerting * Anomaly Detection * Centralized Log >>>> >> > >> Management >>>> >> > >> > >> > Solr & Elasticsearch Support * http://sematext.com/ >>>> >> > >> > >> >>>> >> > >> > >>>> >> > >> >>>> >> > > >>>> >> > > >>>> >> > > >>>> >> > > -- >>>> >> > > Best regards, >>>> >> > > >>>> >> > > - Andy >>>> >> > > >>>> >> > > Problems worthy of attack prove their worth by hitting back. - Piet >>>> >> Hein >>>> >> > > (via Tom White) >>>> >> > >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> Best regards, >>>> >> >>>> >> - Andy >>>> >> >>>> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein >>>> >> (via Tom White) >>>> >> >>>> >>> >>> >>> >>> -- >>> Best regards, >>> >>> - Andy >>> >>> Problems worthy of attack prove their worth by hitting back. - Piet Hein >>> (via Tom White) > > > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal
