GitHub user mlangc added a comment to the discussion: Call for feedback: I've implemented an AsyncHttpAppender
> Sorry for the delayed response, and thanks for sharing your work: it’s great > to see experimentation around this area. Thanks a lot for your feedback! > Am I correct in assuming that this relies on server-side APIs that can accept > multiple log events per request? Yes, that's correct, all log ingestion APIs that I looked into support batching. The `AsyncHttpAppender` creates batches out of buffered and serialized log events using `batchPrefix`, `batchSuffix` and `batchSeparator` options. > have you already experimented with the asynchronous facilities that are part > of Log4j Core itself Yes, this was one of the first things I did, however the fundamental problem is that pushing logs against an HTTP API one-by-one is inefficient and slow by design. Asynchronous loggers can hide the slowness, but only for very low average log throughputs. Without batching, even one log event per second can add significant overhead. > asynchronous loggers are in use, additional asynchronous behavior inside > appenders can sometimes become more of a liability than a benefit I did my best to make the implementation efficient and robust. The `AsyncHttpAppender` uses a [single lock](https://github.com/mlangc/more-log4j2/blob/1eb67914c7e6da71f623cf23d2e7924345039441/core/src/main/java/com/github/mlangc/more/log4j2/appenders/AsyncHttpAppender.java#L94-L94) that is only held shortly, and all asynchronous operations, including HTTP IO [with retries](https://github.com/mlangc/more-log4j2/blob/1eb67914c7e6da71f623cf23d2e7924345039441/core/src/main/java/com/github/mlangc/more/log4j2/appenders/HttpRetryManager.java#L67-L67), are performed on a [single, dedicated thread](https://github.com/mlangc/more-log4j2/blob/1eb67914c7e6da71f623cf23d2e7924345039441/core/src/main/java/com/github/mlangc/more/log4j2/appenders/AsyncHttpAppender.java#L650-L650). I did [a lot of testing](https://github.com/mlangc/more-log4j2/blob/1eb67914c7e6da71f623cf23d2e7924345039441/core/src/test/java/com/github/mlangc/more/log4j2/appenders/AsyncHttpAppenderTest.java#L406-L406) to make sure that the appender behaves properly, especially under adverse conditions like high concurrency, unreliable backends and extreme loads, but sure, bugs cannot be ruled out. Last but not least, doing synchronous network IO potentially traversing multiple time zones in the logging thread is quite problematic as well. If retries are involved, it might take seconds till a message is finally delivered. During this time, the affected application threads are blocked from doing anything useful. In extreme cases, the entire application might be blocked from making progress. > It introduces an extra async boundary Yes, however only per batch, and not per log line. Normally the [log event is rendered](https://github.com/mlangc/more-log4j2/blob/1eb67914c7e6da71f623cf23d2e7924345039441/core/src/main/java/com/github/mlangc/more/log4j2/appenders/AsyncHttpAppender.java#L355-L355), and [appended to the current batch](https://github.com/mlangc/more-log4j2/blob/1eb67914c7e6da71f623cf23d2e7924345039441/core/src/main/java/com/github/mlangc/more/log4j2/appenders/AsyncHttpAppender.java#L379-L379) synchronously. > especially if the appender implementation performs logging internally I'm only using the `StatusLogger` internally - however there is an advanced batch completion listener option, that could lead to logs being generated from within the appender. This is mentioned explicitly [in the documentation](https://github.com/mlangc/more-log4j2?tab=readme-ov-file#read-this-if-you-want-to-play-with-the-batchcompletionlistener-option). > Remove the asynchronous execution from the HTTP appender itself and rely on > AsyncLogger for offloading work. > You could likely simplify the batching logic by leveraging > [LogEvent#isEndOfBatch()](https://logging.apache.org/log4j/2.x/javadoc/log4j-core/org/apache/logging/log4j/core/LogEvent.html#isEndOfBatch%28%29). So you mean that we could still batch log events, potentially by inspecting `LogEvent#isEndOfBatch`, but send batches synchronously? This is something I can definitely explore, though I would see that as a potential enhancement of the regular `HttpAppender`. Regarding using `LogEvent#isEndOfBatch`: I need to have a deeper understanding about the involved mechanics, but I wonder if the batches resulting from this approach would be appropriate for an HTTP backend with regards to size and the frequency in which they would be delivered. > In the past, we have discussed generalizing > [HttpManager](https://logging.apache.org/log4j/2.x/javadoc/log4j-core/org/apache/logging/log4j/core/appender/HttpManager.html) I like the idea. Ideally this advanced `HttpManager` could then also offer an asynchronous API. This way users could benefit from `java.net.http.HttpClient` if they are on newer Java versions just by adding an additional dependency if I understand the part with the `ServiceLoader` correctly. I'm happy to explore/experiment in this direction and share my findings for discussion if you are interested. GitHub link: https://github.com/apache/logging-log4j2/discussions/4020#discussioncomment-15588883 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
