xiangfu0 opened a new pull request, #19012:
URL: https://github.com/apache/pinot/pull/19012
## Summary
Follow-up to #18972 and @Jackie-Jiang's request to extend the transform
function.
This adds query-time and ingestion-transform equivalents of the new
streaming JsonPath scalar functions:
```text
jsonExtractScalarFast(jsonField, jsonPath, resultsType[, defaultValue])
jsonExtractScalarFirstMatch(jsonField, jsonPath, resultsType[, defaultValue])
```
- `jsonExtractScalarFast` streams simple paths and scans the full root value.
- `jsonExtractScalarFirstMatch` stops as soon as the addressed value is
found.
- Complex paths, unsupported roots, and fast-extractor failures fall back to
the existing Jayway implementation.
- The existing `jsonExtractScalar` execution path remains Jayway-based for
compatibility.
- Both `STRING` and raw JSON `BYTES` inputs are supported.
- Planner registration, operand validation, return-type inference,
function-registry snapshots, and both query engines are covered.
- Adds the missing multi-path `byte[]` extractor overload needed by the
transform path.
## User guide
### Choosing a function
| Function | Use when | Duplicate keys | Malformed content after the match |
| --- | --- | --- | --- |
| `jsonExtractScalar` | Existing behavior is required | Last value wins |
Rejected |
| `jsonExtractScalarFast` | You want streaming extraction with
Jayway-compatible ordinary-input behavior | Last value wins | Rejected |
| `jsonExtractScalarFirstMatch` | Input is known to be well-formed and
duplicate-free, and early exit is useful | First non-null value wins | Not
validated |
The fast path supports simple linear paths: `$` followed by `.key`,
`['key']`, or `[index]`. Wildcards, recursive descent, filters, unions, slices,
negative indices, and other complex paths automatically use Jayway.
As documented for the extractor introduced in #18972, `Fast` may avoid
Jayway materialization failures in an unaddressed subtree, such as an
over-limit string or pathological `BigDecimal` exponent. It does not return a
different addressed value.
### Arguments and result types
- `jsonField` must be a single-value `STRING` or `BYTES` column or transform
expression.
- `jsonPath` and `resultsType` must be string literals.
- `defaultValue`, when supplied, may be any literal.
- Without a default, an unresolved scalar row throws; an unresolved array
becomes an empty array. Null elements inside a resolved array still require a
default.
Supported scalar result types:
```text
INT, LONG, FLOAT, DOUBLE, BIG_DECIMAL, BOOLEAN, TIMESTAMP,
STRING, JSON, BYTES
```
Supported array result types:
```text
INT_ARRAY, LONG_ARRAY, FLOAT_ARRAY, DOUBLE_ARRAY,
BIG_DECIMAL_ARRAY, BOOLEAN_ARRAY, TIMESTAMP_ARRAY, STRING_ARRAY
```
Additional conversion notes:
- `BOOLEAN` accepts booleans, non-zero numbers, and Pinot boolean strings.
- `TIMESTAMP` accepts epoch milliseconds or strings supported by Pinot
timestamp conversion.
- `STRING` preserves strings and serializes other JSON values.
- `JSON` is exposed to SQL as serialized `VARCHAR`.
- `BYTES` decodes a Base64-encoded JSON string.
- `BIG_DECIMAL` extraction preserves decimal precision.
Multi-stage queries support every listed type except `BIG_DECIMAL_ARRAY`;
the planner currently lowers decimal arrays to `DOUBLE_ARRAY`. Use the
single-stage engine when decimal-array precision matters.
## Sample queries
```sql
SELECT
jsonExtractScalarFast(payload, '$.user.id', 'LONG', -1) AS user_id,
jsonExtractScalarFirstMatch(
payload,
'$.service.name',
'STRING',
'unknown'
) AS service_name,
jsonExtractScalarFast(payload, '$.tags', 'STRING_ARRAY') AS tags
FROM events
LIMIT 10;
```
A `BYTES` column containing the raw JSON document works the same way:
```sql
SELECT
jsonExtractScalarFast(payloadBytes, '$.amount', 'BIG_DECIMAL', 0) AS amount
FROM events
LIMIT 10;
```
The functions can also be used in ingestion transforms:
```json
{
"ingestionConfig": {
"transformConfigs": [
{
"columnName": "userId",
"transformFunction": "jsonExtractScalarFast(payload, '$.user.id',
'LONG', -1)"
},
{
"columnName": "serviceName",
"transformFunction": "jsonExtractScalarFirstMatch(payload,
'$.service.name', 'STRING', 'unknown')"
}
]
}
}
```
## Testing
Passed on the rebased commit:
- `FastJsonPathExtractorTest` — 488 tests, including the new multi-path
`byte[]` overload.
- `JsonExtractScalarTransformFunctionTest` — 199 tests across all three
modes, scalar/array types, coercion, defaults, complex-path fallback,
`STRING`/`BYTES` input, Base64 `BYTES` output, duplicate keys, and malformed
tails.
- `FunctionRegistryTest` — 4 tests.
- `QueryCompilationTest` — 221 tests, including planner typing and invalid
non-literal operands.
- `JsonPathTest` — 48 integration tests on both query engines with `STRING`
and `BYTES` JSON columns.
- Affected-module `test-compile` with `-Xlint:all`.
- `spotless:apply`, `checkstyle:check`, `license:format`, and
`license:check` for all affected modules.
The `UdfTest#failWhenAllFunctionsYamlNotUpdated` snapshot comparison is
currently blocked on upstream `master` before reaching its assertion: `NotUdf`
reflects `LogicalFunctions.not(boolean)`, while the current method signature is
`not(Boolean)`. The expected snapshot is updated here with the two transform
registrations and the six scalar registrations merged in #18972.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
