soumya-ghosh commented on PR #1057: URL: https://github.com/apache/iceberg-python/pull/1057#issuecomment-2290024214
@sungwy @kevinjqliu I misunderstood the functioning of `_task_to_record_batches`, thus ended up making unnecessary changes, thanks for review comments. After running some tests I realized how `to_arrow_batch_reader` is different from `to_arrow`. I've updated the logic to handle limit in `_task_to_table` method. Instead of materializing all batches, I'm iterating through batches and stopping early if sufficient records to satisfy the limit clause are read. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
