osscm commented on issue #10335: URL: https://github.com/apache/iceberg/issues/10335#issuecomment-2113659715
Thanks Amogh, For us, it the first problem, (didn’t debug if worker pool is not used for certain tasks, but for scanning it is using) I also have similar idea of passing executor from the engine or can provide the number of threads, we can give both the options. As if I can just trust Iceberg APIs to manage the executor, than will just pass the number of threads. Happy to add the change, as seems to be should be straightforward. Also, do you know if there is some test or analysis done to understand how much memory scan can take based on the snapshots/manifests/data-files? On Tue, May 14, 2024 at 2:09 PM Amogh Jahagirdar ***@***.***> wrote: > yeah we have a planWith API on Scans already and Trino is already using > that when generating the splits here > https://github.com/trinodb/trino/blob/master/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergSplitManager.java#L91 > . > > @osscm <https://github.com/osscm> Do you mind elaborating what kind of > configuration you had in mind? It seems to me that with a custom executor > (whose size is controlled by the engine) a user could control the things > you were talking about like memory consumption. Or if you see that we're > not leveraging that threadpool in certain cases, when we should be but > that's another problem > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/iceberg/issues/10335#issuecomment-2111146848>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AXQ2PYXU32MHSZCMMHNXYGLZCJ4RZAVCNFSM6AAAAABHVWF4MSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJRGE2DMOBUHA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
