GitHub user ppkarwasz added a comment to the discussion: Call for feedback: I've implemented an AsyncHttpAppender
Hi @mlangc, Sorry for the delayed response, and thanks for sharing your work: it’s great to see experimentation around this area. > *[the regular HttpAppender] its performance is not acceptable even for toy > projects* I haven’t benchmarked it myself, but that assessment wouldn’t surprise me. `HttpAppender` is one of those components that was added without a strong, established user base driving its design or evolution, so its limitations aren’t particularly shocking. Your appender looks quite interesting, especially the **batching** support, which directly tackles the main drawback of HTTP: per-request overhead. Am I correct in assuming that this relies on server-side APIs that can accept multiple log events per request? Or do the backends you target simply treat logs as a raw character stream? Regarding the **asynchronous** aspect: have you already experimented with the asynchronous facilities that are part of Log4j Core itself (see [Asynchronous loggers](https://logging.apache.org/log4j/2.x/manual/async.html))? They aim to solve the same problem: minimizing latency on the logging call path. `AsyncLogger` does this by using the LMAX Disruptor to hand off log events to a dedicated worker thread, where all blocking operations occur. When asynchronous loggers are in use, additional asynchronous behavior inside appenders can sometimes become more of a liability than a benefit: * it introduces an extra async boundary, * and in rare cases it can even lead to deadlocks (see apache/logging-log4j2#2893), especially if the appender implementation performs logging internally. ### Improvement proposal It might be worth exploring the following approach: * Remove the asynchronous execution from the HTTP appender itself and rely on `AsyncLogger` for offloading work. * You could likely simplify the batching logic by leveraging [`LogEvent#isEndOfBatch()`](https://logging.apache.org/log4j/2.x/javadoc/log4j-core/org/apache/logging/log4j/core/LogEvent.html#isEndOfBatch%28%29). This flag is set when the Disruptor drains its queue, effectively marking the end of a contiguous batch of events: a natural trigger point for issuing an HTTP request. Overall, this looks like a promising direction, and I’m curious to see how it evolves. In the past, we have discussed generalizing [`HttpManager`](https://logging.apache.org/log4j/2.x/javadoc/log4j-core/org/apache/logging/log4j/core/appender/HttpManager.html) (the engine behind `HttpAppender`) so that alternative implementations could be discovered via `ServiceLoader`. This is still an option we could revisit: it would allow `HttpAppender` to transparently benefit from enhanced implementations whenever your artifact is present on the classpath, without requiring configuration changes from users. GitHub link: https://github.com/apache/logging-log4j2/discussions/4020#discussioncomment-15584631 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
