Comments inline
> On Dec 24, 2019, at 2:09 AM, Volkan Yazıcı <volkan.yaz...@gmail.com> wrote:
>
> This is easier said than done. Customizing the schema and injected
> content is the raison d'être of LogstashLayout. Would you people
> accept a PR re-branding LogstashLayout as the new JsonLayout where the
> default JSON schema conforms with the JsonLayout of Log4j 2.0? This
> would provide 100% backward-compatibility with the existing JsonLayout
> and introduce all the benefits (i.e., configuration knobs, zero TLA)
> of LogstashLayout.
Personally, I would have no problem accepting a contribution that is backwards
compatible and meets ASF requirements; it has to use the Apache License and the
contribution needs to be signed off by anyone who has copyright claims. That
can either be done via one or more ICLAs or via a software grant. If it is just
you then you would just need to sign an ICLA. Also, we would hope you would
stick around to provide support for the component.
>
> @Ralph, do you have any plans on your mind for the API to expose to
> the user for customizing the JSON schema? In LogstashLayout, I use an
> ad-hoc JSON templating -- see Log4j2JsonLayout.json[1], for instance.
> I am not proud of this way of templating, you can't express every kind
> of transformation, e.g., "message": "foo bar baz ${json:message}" will
> not yield the output what one would expect it to. That said, it is
> simple and powerful enough to empower thousands of users so far.
> Further, I have never received any single complaints about its
> shortcomings. (If there are bright minds in the room with some
> JsonT[2] and/or fn:xml-to-json[3], please chime in.)
No, I am just thinking that when storing into ElasticSearch user’s typically
will want the data stored in the manner they want to query it. I know for my
use case I care about the timestamp, formatted message, thread context data and
any event properties. I would want the message portion formatted using a
PatternLayout. I modified the GELF Layout in 2.13.0 to do that so it pretty
much meets my needs to log with Logstash. In fact, the JSON Layout should be
flexible enough that one could configure it to emulate the GELF format.
>
> For XML layout, we can add a configuration knob for XSLT
> transformation that kicks in at rendering time. How to perform this
> "efficiently" (e.g., can we compile the transformation once and reuse
> it as in LogstashLayout) is an open question. We can work that detail
> out in later releases since it will be a user-invisible enhancement.
> (Given both XML and HTML are SGML derivatives, can't we employ the
> same XSLT transformation magic in HtmlLayout design too?)
>
> For YAML layout... Is anybody using it at all? If so, I am really
> curious about the use cases. If there is a demand, we can introduce
> custom schema templating (ala LogstashLayout) to YamlLayout too.
> Though judging from my experience in LogstashLayout, this is quite
> some work. The absence of any YAML schema transformation tools in the
> F/OSS market is also a strong indication that this is not trivial and
> implies many subtle corner cases that needs to be brushed.
As I recall the JSON Layout originally did not use Jackson. I believe Gary
converted it to use it. I don’t recall why. Once it was converted I believe he
decided he could convert the XML Layout and create a YAML Layout almost for
free by leveraging the Jackson support.
>
> I would like to share some remarks about introducing dependencies
> (e.g., Jackson) here.
>
> - For JSON rendering, we can do without Jackson. For
> instance log4j2-ecs-layout[4] renders JSON using its own
> simple single-class-file tooling and *optionally* falls back
> to Jackson for serializing custom Message objects.
>
> - Ever single extra dependency on top of Log4j core
> increments the exponent of dev(ops) nightmare.I find
> myself explaining how to properly package
> log4j2-logstash-layout to dev(ops) people a couple
> of times every year.
As I said, I don’t believe the Layout has to be married to Jackson. As far as
I am concerned, the fewer dependencies the better. In fact, this would mean the
JSONLayout could remain as part of log4j-core in 3.0.
>
> In case you wonder, my intention is to collect feedback, conclude on a
> blueprint blessed by existing contributors, break the blueprint down
> into structured JIRA stories, and start working on them.
>
> [1]
> https://github.com/vy/log4j2-logstash-layout/blob/master/layout/src/test/resources/Log4j2JsonLayout.json
> [2] https://goessner.net/articles/jsont/
> [3] https://www.w3.org/TR/xslt-30/#func-xml-to-json
> [4] https://github.com/elastic/java-ecs-logging/tree/master/log4j2-ecs-layout
>
Ralph
