Oh yes, I’d still love to see how we can adapt the Log4j plugin system here! 
And yeah, this would likely make sense as its own repository. I’ll make a 
proposal about the OTel stuff before working on any code.
—
Matt Sicker

> On Nov 29, 2023, at 22:54, Ralph Goers <ralph.go...@dslextreme.com> wrote:
> 
> This is a great post Matt!
> 
> Fluentbit, Fluentd, Logstash, and Filebeat are the main tools used for log 
> forwarding. While they all have some amount of plugability none of the are as 
> flexible as Flume. In addition, as I have mentioned before, none of them 
> provide guaranteed delivery so I would never recommend them for forwarding 
> audit logs. 
> 
> I have also previously explained my use case for using Flume, which is for 
> forwarding Call Detail Records that start off as records in the Radius 
> protocol [1] across data centers, which also requires guaranteed delivery. I 
> wouldn’t be able to use any of those other tools to do that without 
> significant modification.
> 
> I am all for supporting standards. If you can outline what you are proposing 
> on a Confluence page I would wholeheartedly support it.
> 
> As you probably know, I started work on separating out things that I don’t 
> consider to be “core” to Flume into separate repos. That work is only half 
> completed. I would suggest that you consider whether what you are proposing 
> also be in its own repo. As it is, the CI for Flume fails because the 
> generated logs are exceeding the available disk space. In addition, the build 
> takes a long time. 
> 
> Also, I have never really been a big fan of the configuration mechanism Flume 
> uses. I was able to somewhat bypass it by implementing support for Spring 
> Boot, but it would be great if the Log4j Plugin system could somehow be used 
> to simplify configuring Flume for those who don’t want to use Spring boot. I 
> know that is right up your alley.
> 
> Ralph
> 
> 
> 1. https://networkradius.com/doc/current/introduction/RADIUS.html
> 
>> On Nov 29, 2023, at 5:32 PM, Matt Sicker <m...@musigma.org> wrote:
>> 
>> One of the main reasons why I supported Flume joining this PMC was that I 
>> noticed it has significant overlap with projects in the observability space 
>> despite not being advertised as such. For example, the project FluentBit is 
>> extremely similar to Flume, but its main purpose is for collecting, 
>> processing, forwarding, etc., logs, metrics, and traces (i.e., observability 
>> data). FluentBit is not the only thing in this space, though it seems to be 
>> fairly popular. These sorts of tools are used for ultimately publishing 
>> observability data to one or more observability tools like Prometheus, 
>> Splunk, Jaeger, Grafana, etc., and with a unified collector and processor, 
>> it becomes possible to publish all your observability data into one tool 
>> rather than three or more disparate tools (and the added operational costs 
>> of storing tons of duplicated log data from three or more methods of 
>> generating log data).
>> 
>> A project at the CNCF, OpenTelemetry, has become the sort of de facto 
>> standard for interoperability in this space. In particular, they’ve 
>> published the OTLP specification <https://opentelemetry.io/docs/specs/otlp/> 
>> for general telemetry data delivery and the OpenTelemetry specification 
>> <https://opentelemetry.io/docs/specs/otel/> for various common APIs. While 
>> I’m still researching in this space, I think it would be useful for Flume to 
>> integrate with some of these APIs and SDKs (while other parts might be more 
>> relevant in our logging libraries instead). There is also the Open Agent 
>> Management Protocol <https://opentelemetry.io/docs/specs/opamp/> which is 
>> still in beta status that might also be relevant here (and potentially 
>> relevant in the logging libraries).
>> 
>> Supporting common standards for our projects seems like a useful thing to 
>> do, and despite the popularity of some existing solutions there, I believe 
>> there is plenty of space for us to contribute. I also think that this can 
>> provide opportunity for the various components in this PMC to interoperate 
>> as these specs are fairly language neutral with some sample versions in many 
>> different languages.
> 
> 

Reply via email to