Delaying publication of `log4j-flume-ng` in 3.x
Hi all, The `log4j-flume-ng` module _de facto_ contains 3 different appenders: * an Avro appender, that only depends on Avro and Avro IPC. Since it only communicates with Flume via network, it doesn't need to even depend on Flume, it just needs to use the same protocol. * a Persistent appender, which works the same way as Avro appender, but stores the messages to a local database first. It has an additional dependency on `je` (a mostly unmaintained implementation of BerkeleyDB). * an Embedded appender, which is the only one that actually depends on Flume. It also contains an undocumented (and untested) `Log4jEventSource` plugin for Flume. The use of optional dependencies is not very JPMS friendly and we probably would need to split it into 3 modules. To prevent delays in releasing 3.0.0 I would propose to: * either move the `log4j-flume-ng` module to a separate repo and release version `3.0.0` together with the next Flume release. * or move it from the `main` branch to a different branch that will be merged, when the module is ready. What do you think? Piotr
Re: Delaying publication of `log4j-flume-ng` in 3.x
I support the idea of removing `log4j-flume-ng` from `main` and figuring out a resolution later. I think it is too early to decide on how to further proceed with `log4j-flume-ng`; moving to a separate repository, merging it back to `main`, etc. The important thing is, we can always add it back to `main`. On a similar matter, I am inclined to move `log4j-flume-ng` (in `2.x`) to a separate repository. It accounts for 10% of the build time. It is a very stable product with a happy user base. There is no need to keep its release cadence chained to `2.x`. On Thu, Aug 29, 2024 at 2:12 PM Piotr P. Karwasz wrote: > Hi all, > > The `log4j-flume-ng` module _de facto_ contains 3 different appenders: > > * an Avro appender, that only depends on Avro and Avro IPC. Since it > only communicates with Flume via network, it doesn't need to even > depend on Flume, it just needs to use the same protocol. > * a Persistent appender, which works the same way as Avro appender, > but stores the messages to a local database first. It has an > additional dependency on `je` (a mostly unmaintained implementation of > BerkeleyDB). > * an Embedded appender, which is the only one that actually depends on > Flume. > > It also contains an undocumented (and untested) `Log4jEventSource` > plugin for Flume. > > The use of optional dependencies is not very JPMS friendly and we > probably would need to split it into 3 modules. To prevent delays in > releasing 3.0.0 I would propose to: > > * either move the `log4j-flume-ng` module to a separate repo and > release version `3.0.0` together with the next Flume release. > * or move it from the `main` branch to a different branch that will be > merged, when the module is ready. > > What do you think? > > Piotr >
Re: Delaying publication of `log4j-flume-ng` in 3.x
Hi, I'd like to follow Ralph's lead on Flune related topics. Gary On Thu, Aug 29, 2024, 8:12 AM Piotr P. Karwasz wrote: > Hi all, > > The `log4j-flume-ng` module _de facto_ contains 3 different appenders: > > * an Avro appender, that only depends on Avro and Avro IPC. Since it > only communicates with Flume via network, it doesn't need to even > depend on Flume, it just needs to use the same protocol. > * a Persistent appender, which works the same way as Avro appender, > but stores the messages to a local database first. It has an > additional dependency on `je` (a mostly unmaintained implementation of > BerkeleyDB). > * an Embedded appender, which is the only one that actually depends on > Flume. > > It also contains an undocumented (and untested) `Log4jEventSource` > plugin for Flume. > > The use of optional dependencies is not very JPMS friendly and we > probably would need to split it into 3 modules. To prevent delays in > releasing 3.0.0 I would propose to: > > * either move the `log4j-flume-ng` module to a separate repo and > release version `3.0.0` together with the next Flume release. > * or move it from the `main` branch to a different branch that will be > merged, when the module is ready. > > What do you think? > > Piotr >
Re: Delaying publication of `log4j-flume-ng` in 3.x
I’d be ok with splitting it out. > On Aug 29, 2024, at 07:12, Piotr P. Karwasz wrote: > > Hi all, > > The `log4j-flume-ng` module _de facto_ contains 3 different appenders: > > * an Avro appender, that only depends on Avro and Avro IPC. Since it > only communicates with Flume via network, it doesn't need to even > depend on Flume, it just needs to use the same protocol. > * a Persistent appender, which works the same way as Avro appender, > but stores the messages to a local database first. It has an > additional dependency on `je` (a mostly unmaintained implementation of > BerkeleyDB). > * an Embedded appender, which is the only one that actually depends on Flume. > > It also contains an undocumented (and untested) `Log4jEventSource` > plugin for Flume. > > The use of optional dependencies is not very JPMS friendly and we > probably would need to split it into 3 modules. To prevent delays in > releasing 3.0.0 I would propose to: > > * either move the `log4j-flume-ng` module to a separate repo and > release version `3.0.0` together with the next Flume release. > * or move it from the `main` branch to a different branch that will be > merged, when the module is ready. > > What do you think? > > Piotr
Re: Delaying publication of `log4j-flume-ng` in 3.x
My only concern for splitting it out is the same one I have for all of our modules, whether they reside in the same repo as Log4j or not. That is, they all need to be tested against the latest versions of Log4j. In the case of the Flume appender unless something specifically is being done to it there are only 2 reasons it needs a new release: 1. It needs to be updated for a new version of Flume 2. It needs to be updated for something Log4j requires. We know that for 3.0 we changed how Plugins bind with Log4j Core. While 2.x bindings should still work I would prefer that ALL Log4j provided plugins natively support 3.x. All that is to say, all of this could be done with the Flume Appender residing in its own repo. It could either provide its own 3.x branch or provide a log4j-flume-3x jar. I kind of doubt it but did we make it so a plugin could be built for both 2.x and 3.x at the same time? I.e. Still provide a log4jplugins.dat file and have the ServiceLoader bindings? So I will say I am fine with moving it to its own module. Can it be moved while keeping the 2.x and main branches? I looked at the Nexus repo stats and was actually surprised that log4j-flume-ng falls smack in the middle of the pack. That said, 2/3 of our artifacts show up as 0% - they are dwarfed by these: log4j-bom41,694,665.00 25.80% log4j-api 38,197,952.00 23.64% log4j-core21,092,891.00 13.05% log4j-to-slf4j17,363,471.00 10.75% log4j13,990,810.00.8.66% log4j-slf4j-impl 9,797,055.00 6.06% log4j-1.2-api 4,452,489.00 2.76% log4j-layout-template-json 4,393,231.00 2.72% log4j-jul 2,799,082.00.1.73% log4j-slf4j2-impl 2,695,794.00.1.67% log4j-web 2,367,956.00.1.47% log4j-appserver745,912.00. 0.46% log4j-jcl 606,258.00 0.38% log4j-iostreams185,659.00 0.11% log4j-slf4j18-impl 173,961.00 0.11% log4j-spring-boot 158,438.00 0.10% However, usage of the Flume Appender appears to be greater than some others where you might not think it would be log4j-flume-ng 10,367.00 0.01% log4j-taglib 10,236.00 0.01% log4j-spring-cloud-config 9,769.00 0.01% log4j-couchdb9,331.000.01% log4j-jmx-gui8,345.000.01% log4j-liquibase 5,903.000.00% log4j-jakarta-smtp 4,201.000.00% log4j-mongodb4 3,741.00 0.00% log4j-jpa2,507.000.00% log4j-mongodb3 1,047.000.00% Ralph > On Aug 29, 2024, at 5:12 AM, Piotr P. Karwasz wrote: > > Hi all, > > The `log4j-flume-ng` module _de facto_ contains 3 different appenders: > > * an Avro appender, that only depends on Avro and Avro IPC. Since it > only communicates with Flume via network, it doesn't need to even > depend on Flume, it just needs to use the same protocol. > * a Persistent appender, which works the same way as Avro appender, > but stores the messages to a local database first. It has an > additional dependency on `je` (a mostly unmaintained implementation of > BerkeleyDB). > * an Embedded appender, which is the only one that actually depends on Flume. > > It also contains an undocumented (and untested) `Log4jEventSource` > plugin for Flume. > > The use of optional dependencies is not very JPMS friendly and we > probably would need to split it into 3 modules. To prevent delays in > releasing 3.0.0 I would propose to: > > * either move the `log4j-flume-ng` module to a separate repo and > release version `3.0.0` together with the next Flume release. > * or move it from the `main` branch to a different branch that will be > merged, when the module is ready. > > What do you think? > > Piotr
Re: Delaying publication of `log4j-flume-ng` in 3.x
Hi Ralph, On Thu, 29 Aug 2024 at 23:32, Ralph Goers wrote: > So I will say I am fine with moving it to its own module. Can it be moved > while keeping the 2.x and main branches? Sure, I created a `logging-log4j-flume` repository and I'll move both branches there preserving the commit history. Piotr
