morningman commented on code in PR #68874:
URL: https://github.com/apache/doris/pull/68874#discussion_r4257182885
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertCommandContext.java:
##########
@@ -84,16 +91,69 @@ public void
setStaticPartitionSpecFromExpressions(Map<String, Expression> partit
// Commit metadata must use the same semantic bytes as the
materialized partition row.
literal = (Literal)
ConnectorWriteValueConverter.convert(column, literal);
}
- // Binary partition keys must bypass character decoding,
including empty and non-UTF-8 bytes.
- spec.put(entry.getKey(), literal instanceof VarBinaryLiteral
- ? "0x" + literal.toString() :
literal.getStringValue());
+ spec.put(entry.getKey(), specValue(literal));
+ literals.put(entry.getKey(), literal);
if (entry.getValue() instanceof NullLiteral) {
nullKeys.add(entry.getKey());
}
}
}
this.staticPartitionSpec = spec;
this.staticPartitionNullKeys = nullKeys;
+ this.staticPartitionLiterals = literals;
+ }
+
+ public void setBoundTargetSchema(List<Column> boundTargetSchema) {
+ this.boundTargetSchema = boundTargetSchema;
+ }
+
+ /**
+ * Casts the static partition values to the types of their columns in the
bound target schema; see
+ * {@code ConnectorWriteHandle#getCastStaticPartitionSpec}. Each value is
the one BindSink materializes into
+ * the rows, after constant folding, but a value that does not fit its
type fails here instead of becoming
+ * NULL.
+ */
+ public Map<String, String> castStaticPartitionSpec() {
+ Preconditions.checkState(staticPartitionLiterals.isEmpty() ||
!boundTargetSchema.isEmpty(),
+ "static partition values are cast before the sink is bound to
a target schema");
+ Map<String, String> spec = new LinkedHashMap<>();
+ for (Map.Entry<String, Literal> entry :
staticPartitionLiterals.entrySet()) {
+ if (staticPartitionNullKeys.contains(entry.getKey())) {
+ continue;
+ }
+ Literal value = entry.getValue();
+ Optional<Column> column = boundTargetSchema.stream()
+ .filter(candidate ->
candidate.getName().equalsIgnoreCase(entry.getKey()))
+ .findFirst();
+ if (column.isPresent()) {
+ value = writtenValue(value,
DataType.fromCatalogType(column.get().getType()));
+ }
+ spec.put(entry.getKey(), specValue(value));
+ }
+ return spec;
+ }
+
+ private static String specValue(Literal literal) {
+ // Binary partition keys must bypass character decoding, including
empty and non-UTF-8 bytes.
+ return literal instanceof VarBinaryLiteral ? "0x" + literal.toString()
: literal.getStringValue();
+ }
+
+ private static Literal writtenValue(Literal value, DataType columnType) {
+ if (!value.getDataType().isStringLikeType() ||
!columnType.isStringLikeType()) {
+ Expression cast = value.checkedCastTo(columnType);
Review Comment:
Thanks. This mismatch needs `debug_skip_fold_constant=true`, and I'd rather
not special-case that switch, so no change here.
In any other session the value BindSink writes into the rows is folded on
FE: `TypeCoercionUtils.castIfNotSameType` leaves a `Cast` over the literal
(DECIMAL to FLOAT and INT to VARCHAR are not promotions), and
`FoldConstantRuleOnFE.visitCast` folds it with the same `Literal.checkedCastTo`
that `castStaticPartitionSpec` calls. Rows and commit metadata therefore agree:
`0.5000000298023223876953125001` to FLOAT is `0.50000006` in both, and `123` to
VARCHAR(2) is `12` in both. `enable_fold_constant_by_be` (fuzzed in CI) does
not change this, since `FoldConstantRule` runs the FE rule first and hands BE
only what FE could not fold. A cast that FE cannot fold makes `checkedCastTo`
throw in `castStaticPartitionSpec` as well (a pair `CheckCast` rejects fails
analysis), so the statement fails rather than writing another partition.
`debug_skip_fold_constant` turns all FE constant folding off
(`FoldConstantRule.evaluateOrThrow` returns the expression as is, and the
pattern rules skip it); it is a debugging switch and is not fuzzed. As you
note, a populated overwrite still fails Paimon's outside-target check, so what
remains is a runtime-empty overwrite in a session that has folding disabled for
debugging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
