MonkeyCanCode commented on PR #14590: URL: https://github.com/apache/iceberg/pull/14590#issuecomment-6667045527
> > I think we should go with the approach presented in #18378 > > Thanks for the confirmation. I will try out the private build locally later tonight and report back if issue persisted. Once confirm resolved, I will close both this PR and the one from Spark side. Hello @nastra , With the purposed changes in https://github.com/apache/iceberg/pull/18378, I am still seeing spark not released the resources: <img width="1365" height="374" alt="image" src="https://github.com/user-attachments/assets/e04e6a28-10cd-482c-80a6-c0bb9fc59818"; /> <img width="1365" height="370" alt="image" src="https://github.com/user-attachments/assets/8abcfbd8-55ed-46eb-928d-6b80cd7710f5"; /> <img width="1365" height="32" alt="image" src="https://github.com/user-attachments/assets/070d2719-bd1b-4441-a156-9116e56adbe0"; /> <img width="1365" height="32" alt="image" src="https://github.com/user-attachments/assets/076cb9aa-c258-4cf1-b920-27d529079e07"; /> This is ran with reproducible I provided in https://github.com/apache/spark/pull/59952 where I set TTL of session to be 1min and each of them do a simple `SELECT * FROM quickstart_table`. Upon some debugging, the fixed from the https://github.com/apache/iceberg/pull/18378 is correct where it clsoed the `MoreExecutors$Application` hook lambdas (only 3 left). However, the released Spark Connect sessions are still retained. This appears to be related to`RESTMetricsReporter` which is default to true in 1.12.0 and each `RESTSessionCatalog` creates its own `rest-metrics-reporter` executor. With `rest-metrics-reporting-enabled=false`, the sessions are no longer retained. I think we will need the change from fix from the https://github.com/apache/iceberg/pull/18378 along with additional change such as lets its idle thread exit by building the pool with allowCoreThreadTimeout and a short keep-alive. What do you think @nastra @adutra @pvary -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
