Re: Revisiting the binary logging format idea

2017-10-21 Thread Mikael Ståldal
On 2017-10-20 06:36, Matt Sicker wrote: On 19 October 2017 at 14:05, Mikael Ståldal wrote: And don't forget a corresponding LogEventParser. Right. The basic idea here would be to provide both a reference encoder and decoder. If an existing binary format such as BSON, Avro, or Thrift were ap

Re: Revisiting the binary logging format idea

2017-10-19 Thread Matt Sicker
On 19 October 2017 at 14:05, Mikael Ståldal wrote: > And don't forget a corresponding LogEventParser. Right. The basic idea here would be to provide both a reference encoder and decoder. If an existing binary format such as BSON, Avro, or Thrift were appropriate, then there wouldn't be much nee

Re: Revisiting the binary logging format idea

2017-10-19 Thread Mikael Ståldal
And don't forget a corresponding LogEventParser. On 2017-10-19 18:56, Remko Popma wrote: I agree with Gary: this is just another Layout. Should not need another repo... Prototyping on another branch makes sense. On Fri, Oct 20, 2017 at 1:23 AM, Gary Gregory wrote: Bleh, ANOTHER repo? We hav

Re: Revisiting the binary logging format idea

2017-10-19 Thread Matt Sicker
The idea behind this isn't specific to log4j, hence why I suggested a repo. Although if none of the projects outside log4j want to use it, then it could simply be a module in log4j or part of log4j-core. On 19 October 2017 at 12:09, Ralph Goers wrote: > Core isn’t a multi-module project so I don

Re: Revisiting the binary logging format idea

2017-10-19 Thread Ralph Goers
Core isn’t a multi-module project so I don’t know how you can have a new module in it. That said, I’m fine with a new module for BSON. Ralph > On Oct 19, 2017, at 10:07 AM, Gary Gregory wrote: > > Related: I was thinking of creating a new _module_ in core called > log4j-bson to provide a BSON

Re: Revisiting the binary logging format idea

2017-10-19 Thread Gary Gregory
Related: I was thinking of creating a new _module_ in core called log4j-bson to provide a BSON Layout. This would be a separate module to only drag in the BSON library when needed. Part of the breaking up Core epic. Gary On Thu, Oct 19, 2017 at 10:56 AM, Remko Popma wrote: > I agree with Gary:

Re: Revisiting the binary logging format idea

2017-10-19 Thread Remko Popma
I agree with Gary: this is just another Layout. Should not need another repo... Prototyping on another branch makes sense. On Fri, Oct 20, 2017 at 1:23 AM, Gary Gregory wrote: > Bleh, ANOTHER repo? We have so many already... but I see what the big > picture is. Would this add a new layout in Cor

Re: Revisiting the binary logging format idea

2017-10-19 Thread Gary Gregory
Bleh, ANOTHER repo? We have so many already... but I see what the big picture is. Would this add a new layout in Core? Maybe we should just start with that... then grow... Gary On Thu, Oct 19, 2017 at 9:57 AM, Matt Sicker wrote: > I mean in the logging services project. So it'd be a new repo. >

Re: Revisiting the binary logging format idea

2017-10-19 Thread Matt Sicker
I mean in the logging services project. So it'd be a new repo. On 19 October 2017 at 10:48, Ralph Goers wrote: > When you say “another component in the project” do you mean logging > services project or log4j? I’d prefer to see you do this in a separate repo > or at least a branch until we under

Re: Revisiting the binary logging format idea

2017-10-19 Thread Ralph Goers
When you say “another component in the project” do you mean logging services project or log4j? I’d prefer to see you do this in a separate repo or at least a branch until we understand what it looks like. If it is going to apply to many things it probably makes sense to be a separate repo. Ralp

Re: Revisiting the binary logging format idea

2017-10-19 Thread Matt Sicker
For generic structured records, I'd probably go with Avro or Thrift since LogEvents have a lot of standard fields with only a few optional map-like structures. For optimized log appending, the binary format was proposed as a way to append quickly and without garbage IIRC. On 19 October 2017 at 10:

Re: Revisiting the binary logging format idea

2017-10-19 Thread Gary Gregory
What about BSON? Gary On Oct 19, 2017 08:41, "Matt Sicker" wrote: > I don't have the ticket on hand, but a few months ago, Remko suggested a > binary logging format that would allow for super fast appends of > log-specific information along with companion files for additional metadata > not com

Revisiting the binary logging format idea

2017-10-19 Thread Matt Sicker
I don't have the ticket on hand, but a few months ago, Remko suggested a binary logging format that would allow for super fast appends of log-specific information along with companion files for additional metadata not commonly used in log messages. I've been thinking about this idea a bit in relati