Oh yes, I’d still love to see how we can adapt the Log4j plugin system here! And yeah, this would likely make sense as its own repository. I’ll make a proposal about the OTel stuff before working on any code. — Matt Sicker
> On Nov 29, 2023, at 22:54, Ralph Goers <ralph.go...@dslextreme.com> wrote: > > This is a great post Matt! > > Fluentbit, Fluentd, Logstash, and Filebeat are the main tools used for log > forwarding. While they all have some amount of plugability none of the are as > flexible as Flume. In addition, as I have mentioned before, none of them > provide guaranteed delivery so I would never recommend them for forwarding > audit logs. > > I have also previously explained my use case for using Flume, which is for > forwarding Call Detail Records that start off as records in the Radius > protocol [1] across data centers, which also requires guaranteed delivery. I > wouldn’t be able to use any of those other tools to do that without > significant modification. > > I am all for supporting standards. If you can outline what you are proposing > on a Confluence page I would wholeheartedly support it. > > As you probably know, I started work on separating out things that I don’t > consider to be “core” to Flume into separate repos. That work is only half > completed. I would suggest that you consider whether what you are proposing > also be in its own repo. As it is, the CI for Flume fails because the > generated logs are exceeding the available disk space. In addition, the build > takes a long time. > > Also, I have never really been a big fan of the configuration mechanism Flume > uses. I was able to somewhat bypass it by implementing support for Spring > Boot, but it would be great if the Log4j Plugin system could somehow be used > to simplify configuring Flume for those who don’t want to use Spring boot. I > know that is right up your alley. > > Ralph > > > 1. https://networkradius.com/doc/current/introduction/RADIUS.html > >> On Nov 29, 2023, at 5:32 PM, Matt Sicker <m...@musigma.org> wrote: >> >> One of the main reasons why I supported Flume joining this PMC was that I >> noticed it has significant overlap with projects in the observability space >> despite not being advertised as such. For example, the project FluentBit is >> extremely similar to Flume, but its main purpose is for collecting, >> processing, forwarding, etc., logs, metrics, and traces (i.e., observability >> data). FluentBit is not the only thing in this space, though it seems to be >> fairly popular. These sorts of tools are used for ultimately publishing >> observability data to one or more observability tools like Prometheus, >> Splunk, Jaeger, Grafana, etc., and with a unified collector and processor, >> it becomes possible to publish all your observability data into one tool >> rather than three or more disparate tools (and the added operational costs >> of storing tons of duplicated log data from three or more methods of >> generating log data). >> >> A project at the CNCF, OpenTelemetry, has become the sort of de facto >> standard for interoperability in this space. In particular, they’ve >> published the OTLP specification <https://opentelemetry.io/docs/specs/otlp/> >> for general telemetry data delivery and the OpenTelemetry specification >> <https://opentelemetry.io/docs/specs/otel/> for various common APIs. While >> I’m still researching in this space, I think it would be useful for Flume to >> integrate with some of these APIs and SDKs (while other parts might be more >> relevant in our logging libraries instead). There is also the Open Agent >> Management Protocol <https://opentelemetry.io/docs/specs/opamp/> which is >> still in beta status that might also be relevant here (and potentially >> relevant in the logging libraries). >> >> Supporting common standards for our projects seems like a useful thing to >> do, and despite the popularity of some existing solutions there, I believe >> there is plenty of space for us to contribute. I also think that this can >> provide opportunity for the various components in this PMC to interoperate >> as these specs are fairly language neutral with some sample versions in many >> different languages. > >
