merlimat opened a new pull request, #25542:
URL: https://github.com/apache/pulsar/pull/25542
### Motivation
The default JSON log format in Pulsar emits Elastic Common Schema (ECS). ECS
is well-defined but Elastic-specific: outside the Elastic stack, downstream
tools have to translate field names before they can use the output.
OpenTelemetry is a vendor-neutral CNCF standard for observability data, and
its log data model + semantic conventions are understood natively by the OTel
Collector (`filelog` receiver), Grafana Loki, Datadog, Splunk, New Relic,
Elastic, and most modern observability backends. Switching the default format
aligns with where the ecosystem is heading:
- **Vendor neutrality.** OTel is governed by a broad industry consortium;
ECS is governed by one vendor.
- **Broader out-of-the-box compatibility.** Flat OTel JSON ingests directly
into most log backends with no field-mapping stage.
- **Trace/log correlation.** `TraceId` / `SpanId` are top-level fields in
the OTel log data model and match what OTel tracing emits — once Pulsar emits
OTel traces, logs and traces correlate automatically in any OTel-aware backend.
- **Forward-looking.** New observability tooling is increasingly built
OTel-first; ECS adoption is flat outside the Elastic ecosystem.
Nothing was broken with ECS — this is a format-alignment change that makes
the default output useful in more places with fewer transformations. Operators
who still want ECS can opt back in with
`PULSAR_LOG_JSON_TEMPLATE=classpath:EcsLayout.json` (the ECS template continues
to ship inside `log4j-layout-template-json`).
### Modifications
- **New** `conf/OtelLogLayout.json` — a Log4j2 `JsonTemplateLayout` template
producing one event per line with the flat OTel shape:
- Top-level: `Timestamp` (ISO-8601 UTC), `SeverityText`, `Body`,
`TraceId`, `SpanId`, `TraceFlags` (trace context read from MDC keys `trace_id`
/ `span_id` / `trace_flags`, the keys OTel's SLF4J MDC integration emits).
- `Attributes`: `thread.name`, `thread.id`, `code.namespace`,
`exception.type` / `exception.message` / `exception.stacktrace`, plus any other
MDC entries flattened in.
- Empty fields are dropped (default `JsonTemplateLayout` behavior), so
trace-context keys don't appear when tracing is inactive.
- Stack traces are stringified for readability and smaller wire size; this
is the OTel-convention shape for `exception.stacktrace`.
- **`conf/log4j2.yaml`** — default `pulsar.log.json.template` changed from
`classpath:EcsLayout.json` to `file:conf/OtelLogLayout.json`.
- **`bin/pulsar`, `bin/bookkeeper`, `bin/pulsar-perf`,
`bin/pulsar-admin-common.sh`, `bin/pulsar-admin-common.cmd`** — default value
of `PULSAR_LOG_JSON_TEMPLATE` / `BOOKIE_LOG_JSON_TEMPLATE` changed to
`file:$PULSAR_HOME/conf/OtelLogLayout.json` (absolute; `pulsar-admin-common.sh`
does not `cd $PULSAR_HOME` before launching, so a relative `file:conf/...`
would break `pulsar-admin` invocations from arbitrary CWDs).
- **`bin/pulsar`, `conf/pulsar_env.sh`** — updated stale "ECS JSON" comments.
Distribution packaging required no change:
`distribution/server/build.gradle.kts` already copies the whole `conf/`
directory into the server tarball.
### Verifying this change
This change is a trivial rework without new test coverage — it swaps one
JSON template for another. Verified locally that:
- `classpath:EcsLayout.json` still works as an opt-out for anyone who needs
ECS.
- `conf/OtelLogLayout.json` parses cleanly as Log4j2 `JsonTemplateLayout`
(runs a broker with `PULSAR_LOG_FORMAT=json`).
### Does this pull request potentially affect one of the following parts:
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [x] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
The default JSON log format changes for users who have set
`PULSAR_LOG_FORMAT=json`. The text log format is the default and is unaffected.
ECS can be restored by setting
`PULSAR_LOG_JSON_TEMPLATE=classpath:EcsLayout.json`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
