hubgeter opened a new pull request, #70509: URL: https://github.com/apache/doris/pull/70509
### Background Iceberg metadata tables expose typed partition values and readable file metrics. The Java metadata scanner carries each field's Iceberg type into the JNI vector adapter, including nanosecond timestamps without a time zone. ### What problem does this PR solve? Issue Number: None Related PR: #68934 (nanosecond type support), #68985 (microsecond metadata timestamp repair), #69163 (nanosecond support on branch-4.2) PR Type: Bugfix Problem Summary: Format-version 3 tables with `timestamp_ns` can return incorrect typed values from metadata tables such as `$files.readable_metrics` and `$partitions.partition`. The adapter interprets nanosecond Long values as microseconds and applies the session time zone to nanosecond NTZ values. For example, `1234567890ns` becomes `00:20:34.567890` instead of `00:00:01.234567890` in UTC; Asia/Shanghai adds a further eight-hour shift, and modern dates can overflow during vector encoding. Decode the value using its actual timestamp type and recognize both NTZ timestamp types when choosing the wall-clock encoding. ### Details - **Frequency:** Whenever a selected, non-NULL nanosecond metadata value reaches this adapter; UTC sessions still have the unit error. - **Root cause:** `timestampInstant()` always uses microsecond units, while `getDateTime()` recognizes only `TimestampType` as NTZ. - **Why existing tests missed it:** Metadata adapter coverage exercised microsecond timestamps. Nanosecond vector coverage supplied already-decoded values and did not exercise the metadata adapter. - **Fix:** Decode `TimestampNanoType` with signed nanosecond units and floor division/modulus; use UTC as the wall-clock encoding convention for both NTZ types. Reuse the existing nested type propagation and vector path. - **Other paths and branches:** Microsecond and untyped-constructor semantics remain covered. This does not enable zoned nanosecond projection. `branch-4.2` at `1bb4a49bddc8a505003d2067cc56f0dc3432f23f` has the same adapter defect after #69163 and needs a backport; its `java-common` interfaces and scanner tests require branch-specific adaptation. - **Behavior change:** NTZ nanosecond metadata preserves all nine fractional digits and does not shift with the session time zone. Valid modern timestamps no longer overflow because of the unit error. ### Testing - Local focused `run-fe-ut.sh` run: **28 passed, 0 failures/errors/skips**. Of these, 24 are tests from committed source (23 metadata scanner tests and one JNI vector test); four additional local reproduction/control checks also passed. `CUSTOM_MVN` scoped Maven's project selection to the metadata scanner while retaining `-am`. - The three original value checks failed before the fix and passed afterward. The added metadata-task tests also reproduced `long overflow` before the fix. - Added coverage includes both session zones, negative nanoseconds, second boundaries, the full signed 64-bit nanosecond range, modern dates, NULL, nested containers, legacy constructors, and serialized FILES/PARTITIONS metadata tasks. - `build.sh --be-java-extensions` with unrelated extensions excluded passed, including Checkstyle for the changed production and test files. - The metadata-task fixtures write real manifests and use metadata-only DataFile entries. They exercise the scanner and off-heap vector within one JVM. Native BE, cross-process JNI, metadata SQL, and remote CI were not run locally. The branch-4.2 assessment is source-based; no branch-4.2 tests were run. ### Release note Fix incorrect values and overflow for Iceberg nanosecond timestamps without a time zone in metadata tables. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Partially verified: local focused tests, Java extension build, and Checkstyle passed; native BE and metadata SQL validation remain outstanding. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Production fix, unit tests, and this description. - Human review: Pending author review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
