OIiveirra commented on code in PR #68161:
URL: https://github.com/apache/doris/pull/68161#discussion_r4080703207
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -192,11 +196,52 @@ private void initReader() throws IOException {
fields.length, paimonAllFieldNames.size()));
}
int[] projected = getProjected();
+ RowType readType = requiresDatetimeV2PrecisionRepair
+ ? createSafeTimestampReadType(table.rowType()) :
table.rowType();
readBuilder.withProjection(projected);
- readBuilder.withFilter(getPredicates());
+ if (requiresDatetimeV2PrecisionRepair) {
+ // withProjection() derives a read type from the table schema and
would otherwise
+ // replace the widened timestamp types with the evolved
(lower-precision) schema.
+ readBuilder.withReadType(readType.project(projected));
+ }
+ readBuilder.withFilter(requiresDatetimeV2PrecisionRepair
+ ? Collections.emptyList() : getPredicates());
reader =
newReadWithOptionalIOManager(readBuilder).executeFilter().createReader(getSplit());
paimonDataTypeList =
- Arrays.stream(projected).mapToObj(i ->
table.rowType().getTypeAt(i)).collect(Collectors.toList());
+ Arrays.stream(projected).mapToObj(i ->
readType.getTypeAt(i)).collect(Collectors.toList());
+ }
+
+ static RowType createSafeTimestampReadType(RowType tableType) {
+ List<DataField> fields = tableType.getFields().stream()
+ .map(field ->
field.newType(createSafeTimestampReadType(field.type())))
+ .collect(Collectors.toList());
+ return tableType.copy(fields);
+ }
+
+ private static DataType createSafeTimestampReadType(DataType dataType) {
+ if (dataType instanceof TimestampType) {
+ return new TimestampType(dataType.isNullable(),
TimestampType.MAX_PRECISION);
+ }
+ if (dataType instanceof LocalZonedTimestampType) {
+ return new LocalZonedTimestampType(dataType.isNullable(),
LocalZonedTimestampType.MAX_PRECISION);
+ }
+ if (dataType instanceof RowType) {
+ RowType rowType = (RowType) dataType;
+ List<DataField> fields = rowType.getFields().stream()
+ .map(field ->
field.newType(createSafeTimestampReadType(field.type())))
+ .collect(Collectors.toList());
+ return rowType.copy(fields);
+ }
+ if (dataType instanceof ArrayType) {
+ ArrayType arrayType = (ArrayType) dataType;
+ return
arrayType.newElementType(createSafeTimestampReadType(arrayType.getElementType()));
+ }
+ if (dataType instanceof MapType) {
+ MapType mapType = (MapType) dataType;
+ return
mapType.newKeyValueType(createSafeTimestampReadType(mapType.getKeyType()),
Review Comment:
Fixed in 90577bc4caa and verified in the latest head. The JNI safe-read
schema preserves MAP key types while recursively widening timestamp
values/elements/row descendants. A reader-level MAP timestamp-key projection
and lookup regression now pass.
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -325,14 +370,18 @@ private Split getSplit() {
private void resetDatetimeV2Precision() {
for (int i = 0; i < types.length; i++) {
- if (types[i].isDateTimeV2()) {
+ if (types[i].isDateTimeV2() || types[i].getType() ==
ColumnType.Type.TIMESTAMPTZ) {
Review Comment:
Fixed in c637de5659b and verified in the latest head. Repair activation now
recursively detects timestamp descendants in ARRAY/ROW/MAP projected columns,
widens the read schema before Paimon materialization, and applies repair after
reading. The nested historical-file projection and predicate regressions pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
