RussellSpitzer commented on issue #8343: URL: https://github.com/apache/iceberg/issues/8343#issuecomment-1814755871
Quick Guess on what might be going wrong, My assumption would be the session being used is not actually loaded with the extensions. I've seen this happen in a few different instances, 1. (Most Common in General) The Spark Session was already created at the time that "getOrCreate" was called and the extensions are ignored. 2. (Most Common in Notebooks) The Spark Session is improperly cloned between threads used by the kernel. I've seen this most commonly with kernels using functional libraries (like cats) or something to managing execution. I'm not sure how this happens (but i've seen in sporadically) but I see that sometimes certain cells will be using SparkSession.getActiveSession to execute their SQL and when they do so they end up picking up a session which somehow was cloned without the config set. When directly queried the config will appear set, but when you access the "active session" during some executions it will vanish. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
