tusharchou commented on code in PR #1388:
URL: https://github.com/apache/iceberg-python/pull/1388#discussion_r1920208710
##########
pyiceberg/table/__init__.py:
##########
@@ -1596,6 +1630,43 @@ def to_ray(self) -> ray.data.dataset.Dataset:
return ray.data.from_arrow(self.to_arrow())
+ def count(self) -> int:
+ # Usage: Calculates the total number of records in a Scan that haven't
had positional deletes.
+ res = 0
+ # every task is a FileScanTask
+ tasks = self.plan_files()
+
+ for task in tasks:
+ # task.residual is a Boolean Expression if the filter condition is
fully satisfied by the
+ # partition value and task.delete_files represents that positional
delete haven't been merged yet
+ # hence those files have to read as a pyarrow table applying the
filter and deletes
+ if task.residual == AlwaysTrue() and not len(task.delete_files):
+ # Every File has a metadata stat that stores the file record
count
+ res += task.file.record_count
+ else:
Review Comment:
@Fokko I have added test of positional delete and it works.
`ArrowScan.to_record_batches` calls `_task_to_record_batches' internally. Can
you please take another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
