Re: Batching in appenders

2020-09-24 Thread Ralph Goers
The more I think about this the more questions I have. As far as I can recall the Appenders that support batching do so because there is a distinct performance benefit to it and the underlying transport mechanism supports it. But almost every different transport handles batching differently. Fo

Re: Batching in appenders

2020-09-24 Thread Ralph Goers
> On Sep 24, 2020, at 11:48 AM, Volkan Yazıcı wrote: > > I think endOfBatch flag is still falling short of addressing the main > issue: the appender is oblivious to the batching. Put another way, it > cannot have an assumption whether endOfBatch is employed or not. Consider > appender A honori

Re: Batching in appenders

2020-09-24 Thread Carter Kozak
We don't provide a great reusable abstraction to batch log events and hand them off to a sink (appender). If we're writing to disk (or similar) this isn't a problem because we buffer at an entirely different layer -- using a RandomAccessFile appender, after passing through the layout, my log eve

Re: Batching in appenders

2020-09-24 Thread Volkan Yazıcı
I think endOfBatch flag is still falling short of addressing the main issue: the appender is oblivious to the batching. Put another way, it cannot have an assumption whether endOfBatch is employed or not. Consider appender A honoring endOfBatch flag, though the preceding filter F never takes advant

Re: Batching in appenders

2020-09-24 Thread Ralph Goers
Yeah, I forgot about the async logger ring buffer. Ralph > On Sep 24, 2020, at 8:56 AM, Carter Kozak wrote: > > The AsyncAppender more or less implements what you've described, except it > forwards each event in the batch individually to a delegate appender and sets > the end-of-batch marker

Re: Batching in appenders

2020-09-24 Thread Carter Kozak
The AsyncAppender more or less implements what you've described, except it forwards each event in the batch individually to a delegate appender and sets the end-of-batch marker on the last one so the API isn't quite as pretty. We could implement something similar that groups events up to some in

Re: Batching in appenders

2020-09-24 Thread Ralph Goers
Well, Log4j only feeds events to Appenders one at a time. It has no place to aggregate them. However, one Appender could wrap others to create a batch at which point this would become useful. I suppose some other component could be invented to create a batch but I am not sure where or how that

Batching in appenders

2020-09-24 Thread Volkan Yazıcı
If one would look close to existing appenders, almost everyone of them implements a custom batching solution. To the best of my knowledge, this is mainly due to the constraint in the Appender interface: void append(LogEvent event); If this would have been replaced (enhanced?) with void a