yadavay-amzn commented on code in PR #16031:
URL: https://github.com/apache/iceberg/pull/16031#discussion_r3112264395
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/CommitMetadata.java:
##########
@@ -56,7 +56,7 @@ public static <R, E extends Exception> R withCommitProperties(
ExceptionUtil.castAndThrow(e, exClass);
return null;
} finally {
- COMMIT_PROPERTIES.set(ImmutableMap.of());
+ COMMIT_PROPERTIES.remove();
Review Comment:
The difference is in ThreadLocal memory management. `set(ImmutableMap.of())`
leaves a live
Entry in the thread's ThreadLocalMap — the slot stays occupied with a
reference to the empty map
`object. remove()` actually removes the Entry from the map, freeing the slot
entirely.
In long-lived executor threads (which Spark reuses across tasks), these
stale entries accumulate and
are never cleaned up since the ThreadLocal is still strongly referenced.
remove() is the recommended
cleanup pattern per the ThreadLocal javadoc
(https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/ThreadLocal.html#remove())
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
