Fokko commented on code in PR #1388:
URL: https://github.com/apache/iceberg-python/pull/1388#discussion_r1950707206
##########
pyiceberg/table/__init__.py:
##########
@@ -1466,6 +1475,25 @@ def _build_partition_evaluator(self, spec_id: int) ->
Callable[[DataFile], bool]
# shared instance across multiple threads.
return lambda data_file: expression_evaluator(partition_schema,
partition_expr, self.case_sensitive)(data_file.partition)
+ def _build_residual_evaluator(self, spec_id: int) -> Callable[[DataFile],
ResidualEvaluator]:
+ spec = self.table_metadata.specs()[spec_id]
+
+ # The lambda created here is run in multiple threads.
+ # So we avoid creating _EvaluatorExpression methods bound to a single
+ # shared instance across multiple threads.
+ # return lambda data_file: (partition_schema, partition_expr,
self.case_sensitive)(data_file.partition)
+ from pyiceberg.expressions.visitors import residual_evaluator_of
+
+ # assert self.row_filter == False
+ return lambda datafile: (
+ residual_evaluator_of(
+ spec=spec,
+ expr=self.row_filter,
+ case_sensitive=self.case_sensitive,
+ schema=self.projection(),
Review Comment:
This should not use the projected schema, but the current schema. It might
be that you're filtering on a column that's not part of the projection.
```suggestion
schema=self.table_metadata.schema(),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
