github-actions[bot] commented on code in PR #65498:
URL: https://github.com/apache/doris/pull/65498#discussion_r3567677221
##########
be/src/format_v2/column_mapper.cpp:
##########
@@ -990,6 +990,18 @@ static bool mapping_can_use_file_column_directly(const
ColumnMapping& mapping) {
return !needs_complex_rematerialize(mapping);
}
+static FilterConversionType direct_filter_conversion(const ColumnMapping&
mapping) {
+ DORIS_CHECK(mapping.table_type != nullptr);
+ // FileScanOperator deliberately keeps VARBINARY predicates above external
readers. Their
+ // physical binary representations are not uniformly supported by
reader-side expression and
+ // metadata filtering, so localizing a late runtime filter here can
incorrectly reject rows.
+ if (remove_nullable(mapping.table_type)->get_primitive_type() ==
TYPE_VARBINARY) {
+ return FilterConversionType::FINALIZE_ONLY;
+ }
+ return mapping.is_trivial ? FilterConversionType::COPY_DIRECTLY
+ : FilterConversionType::CAST_FILTER;
Review Comment:
Please extend this VARBINARY guard to recursive complex dependencies, not
just the top-level primitive mapping. For `s STRUCT<bin VARBINARY>`, the `bin`
child mapping gets `FINALIZE_ONLY`, but the root struct can still be
`COPY_DIRECTLY`/`READER_EXPRESSION`; `localize_filters()` decides from the root
global index and `rewrite_struct_element_path_to_file_expr()` can still push
`element_at(s, 'bin') = ...` into `file_request->conjuncts`. The same
root-type-only check also leaves same-shape `ARRAY<VARBINARY>` or `MAP`
key/value VARBINARY roots local, so generic expressions such as
`array_contains(arr, ...)` or `array_contains(map_keys(m), ...)` can be
localized through `rewrite_table_expr_to_file_expr()`. In all of these cases
the external reader can reject rows using VARBINARY physical binary semantics
before scanner-side final evaluation can correct them. Please make filter
localization detect terminal/nested VARBINARY dependencies before adding
reader-local predicate columns/conju
ncts, with regressions for a VARBINARY struct child and same-shape ARRAY/MAP
VARBINARY predicates. The hidden predicate-only VARBINARY materialization case
is already covered by a live thread in this PR, so I am not duplicating it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
