kevinjqliu commented on issue #1004: URL: https://github.com/apache/iceberg-python/issues/1004#issuecomment-2682383747
Looking at the above, the 2 critical parts are: ``` df=final.arrow() ... tbl.append(df) ``` Im surprised that the `.arrow()` part didn't cause the OOM but the `.append()` part did. > accept recordbatch a valid input ( like delta_rs for example) yep, we got one side of it already. `to_arrow_batch_reader()` produces record batches. We just need to check the write path for `.append()` Its kind of difficult to recreate a scenario where OOM happens. So if you have any other information or a way to reproduce, that'll be very helpful! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
