Mryange opened a new pull request, #70404:
URL: https://github.com/apache/doris/pull/70404
### What problem does this PR solve?
Issue Number: None
Problem Summary:
JSON-to-JSONB conversion constructs a new simdjson On-Demand parser for each
document, repeatedly allocating and freeing its working buffers. Let
`JsonBinaryValue` own and reuse the parser, and move the conversion object
outside both strict and non-strict string-to-JSONB CAST loops. Existing batch
loops in JSON functions, JNI, and Arrow/protobuf SerDe also benefit.
The one-shot parsing API is preserved. Input padding is still allocated per
document. JSONB bytes, storage format, and error/NULL semantics are unchanged;
parser capacity is retained until its owning conversion object is destroyed.
#### Performance
A standalone simdjson v3.11.6 microbenchmark compares **A: construct parser
and padded input per document** with **B: reuse only the parser, still
constructing padded input per document**, matching the lifecycle change in this
PR.
Median ns/document over seven rounds, fully parsing and recursively
consuming every field:
| Synthetic dataset | Average bytes/document | A | B | Time reduction |
Speedup |
| --- | ---: | ---: | ---: | ---: | ---: |
| Short | 103.6 | 219.7 | 181.9 | 17.2% | 1.21x |
| Medium | 447.6 | 370.5 | 311.6 | 15.9% | 1.19x |
| Large | 2855.6 | 1283.0 | 1227.0 | 4.4% | 1.05x |
| Very large | 22119.6 | 8548.4 | 8523.4 | 0.3% | 1.00x |
| Mixed sizes | 6368.1 | 2645.3 | 2569.0 | 2.9% | 1.03x |
Environment: AMD Ryzen 5 9600X, WSL2/Linux, GCC 13.3.0 (`-O3 -DNDEBUG
-std=c++20`), one thread pinned to logical CPU 4; simdjson selected its
`icelake` implementation. Each dataset contains 256 pre-generated documents
with nested objects/arrays, scalar values, and escaped strings. Inputs are
repeatedly cycled in a hot-cache workload; variant order is randomized each
round and checksums are compared. Parser initial allocation is included;
reused-parser destruction occurs after the timed batch. CPU frequency was not
locked.
These results isolate parsing and input-copy costs: they exclude Doris JSONB
encoding, column insertion, and end-to-end query/load execution. They
demonstrate parser-reuse savings, especially for small documents, rather than a
measured Doris throughput improvement. The very-large-document difference is
within timing variation.
### Release note
Reduce repeated parser allocation during batch JSON-to-JSONB conversion,
including string-to-JSONB CAST.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (standalone microbenchmark above)
ASAN BE UT built successfully using `./run-be-ut.sh --run -j48` with a
targeted filter: **85 tests across 7 suites passed**. Coverage includes JSONB
parser/value, strict/non-strict CAST, JSON parsing functions, JSONB SerDe, and
JNI JSON conversion. New cases exercise parser growth, malformed documents,
recovery after errors, and repeated CAST rows. Reused-parser output is compared
against the one-shot parser.
clang-format 16, format checks, build hygiene, and clang-tidy on changed
lines all passed. Full regression tests and end-to-end Doris performance tests
were not run.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
