jimmyxie-figma commented on PR #1677: URL: https://github.com/apache/iceberg-python/pull/1677#issuecomment-2673218581
> @jimmyxie-figma Thanks for raising this PR > > Did you consider just concat'ing the tables: > > ```python > table = pa.concat_tables(tables, promote_options="permissive") > ``` This change was initially intended to improve support for the Ray-Iceberg writer, particularly in cases where a single write operation involves numerous pyarrow tables. However, I realized the correct approach requires supporting an iterator of pyarrow tables, rather than a list, because Ray streams the Arrow tables sequentially from upstream tasks to avoid OOM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
