OIiveirra commented on code in PR #68161:
URL: https://github.com/apache/doris/pull/68161#discussion_r4271830961
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -224,11 +227,11 @@ private void initReader() throws IOException {
: null;
variantProjections.add(variantProjection);
if (variantProjection == null) {
- // The engine may have pruned this complex column down to the
sub-fields the query touches.
- // Mirroring that shape with paimon's own types lets
withReadType push the same projection
- // through the ROW/ARRAY/MAP readers instead of reading the
whole column and discarding it.
DataType projectedType =
PaimonReadTypeProjection.project(tableField.type(),
types[outputIndex]);
+ if (requiresDatetimeV2PrecisionRepair) {
+ projectedType = createSafeTimestampReadType(projectedType);
Review Comment:
Fixed in ff1efbafc860a2fd9fa22b42fd29eee23875d0d7.
QueryAuthSplit rows are materialized at current timestamp precision before
delegating to the original SDK authorization/masking callback. No policy is
dropped or bypassed. Tests cover projected and hidden timestamp policies,
rejected rows, integer-only output, masking and nested values. Timestamp policy
tests use typed SDK callbacks with real historical Parquet files (not
remote-catalog JSON end-to-end coverage).
Verification: 135 focused FE tests and 59 JNI tests passed; full ASAN FE/BE
build and targeted paimon_timestamp_types External regression passed (1 suite,
0 failed/fatal/skipped). Scope remains V2/JNI; V1 is unchanged.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -1289,6 +1306,130 @@ public Map<String, String> getScanNodeProperties(
return props;
}
+ static boolean containsTimestampType(DataType type) {
+ if (type instanceof org.apache.paimon.types.TimestampType
+ || type instanceof
org.apache.paimon.types.LocalZonedTimestampType) {
+ return true;
+ }
+ if (type instanceof org.apache.paimon.types.RowType) {
+ return ((org.apache.paimon.types.RowType)
type).getFields().stream()
+ .anyMatch(field -> containsTimestampType(field.type()));
+ }
+ if (type instanceof org.apache.paimon.types.ArrayType) {
+ return containsTimestampType(((org.apache.paimon.types.ArrayType)
type).getElementType());
+ }
+ if (type instanceof org.apache.paimon.types.MapType) {
+ org.apache.paimon.types.MapType mapType =
(org.apache.paimon.types.MapType) type;
+ return containsTimestampType(mapType.getKeyType()) ||
containsTimestampType(mapType.getValueType());
+ }
+ return false;
+ }
+
+ static boolean filterReadsTimestamp(RowType rowType, ConnectorExpression
filter) {
+ Set<String> names = new HashSet<>();
+ collectFilterColumnNames(filter, names);
+ return rowType.getFields().stream().anyMatch(field ->
+ names.contains(field.name().toLowerCase(Locale.ROOT)) &&
containsTimestampType(field.type()));
+ }
+
+ boolean requiresTimestampPrecisionRepair(ConnectorSession session, Table
table, PaimonTableHandle handle) {
+ if (!containsTimestampType(table.rowType())) {
+ return false;
+ }
+ Supplier<Boolean> lookup = () -> withSchemaHistoryAuthentication(
+ () ->
hasTimestampPrecisionHistory(resolveSchemaDictTable(table, handle)));
Review Comment:
Fixed in ff1efbafc860a2fd9fa22b42fd29eee23875d0d7.
Repair history now resolves the pinned underlying file-store source for
$audit_log, $binlog and $row_tracking, without changing native
schema-dictionary eligibility. FE tests cover all three wrappers and a real
historical $audit_log JNI reader test covers the negative-epoch projection.
Verification: 135 focused FE tests and 59 JNI tests passed; full ASAN FE/BE
build and targeted paimon_timestamp_types External regression passed (1 suite,
0 failed/fatal/skipped). Scope remains V2/JNI; V1 is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
