lkindere opened a new issue, #12792:
URL: https://github.com/apache/iceberg/issues/12792
### Apache Iceberg version
1.7.1
### Query engine
Spark
### Please describe the bug 🐞
Hello,
We have noticed an issue where if a request fails to the catalog while
executing manifest rewrite it will sometimes corrupt the table.
how we think it happens:
SnapshotProducer tries to commit, sends the request to catalog, catalog
registers the request but the response is never received (timeout),
SnapshotProducer attemps to clean up deleting all newly created files, however
the catalog did register the new metadata file and is now missing the files
SnapshotProducer cleaned up
Example:
`RESTException: Error occurred while processing POST request
Caused by: SocketTimeoutException: Read timed out
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:332)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:252)
at org.apache.iceberg.rest.HTTPClient.post(HTTPClient.java:358)
at org.apache.iceberg.rest.RESTClient.post(RESTClient.java:112)
at
org.apache.iceberg.rest.RESTTableOperations.commit(RESTTableOperations.java:152)
at
org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:408)
at
org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at
org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:382)
at
org.apache.iceberg.BaseRewriteManifests.commit(BaseRewriteManifests.java:48)
at
org.apache.iceberg.spark.actions.BaseSnapshotUpdateSparkAction.commit(BaseSnapshotUpdateSparkAction.java:41)
at
org.apache.iceberg.spark.actions.RewriteManifestsSparkAction.replaceManifests(RewriteManifestsSparkAction.java:348)
at
org.apache.iceberg.spark.actions.RewriteManifestsSparkAction.doExecute(RewriteManifestsSparkAction.java:186)
at
org.apache.iceberg.spark.JobGroupUtils.withJobGroupInfo(JobGroupUtils.java:59)
at
org.apache.iceberg.spark.JobGroupUtils.withJobGroupInfo(JobGroupUtils.java:51)
at
org.apache.iceberg.spark.actions.BaseSparkAction.withJobGroupInfo(BaseSparkAction.java:130)
at
org.apache.iceberg.spark.actions.RewriteManifestsSparkAction.execute(RewriteManifestsSparkAction.java:167)
at
wcdp2.maintenance.core.ManifestCompactor$.compact(ManifestCompactor.scala:11)
at
wcdp2.maintenance.core.TableMaintainer$.runCompaction(TableMaintainer.scala:20)
at wcdp2.maintenance.Main$.$anonfun$main$1(Main.scala:23)
at wcdp2.maintenance.Main$.$anonfun$main$1$adapted(Main.scala:22)
at scala.collection.immutable.List.foreach(List.scala:431)
at wcdp2.maintenance.Main$.main(Main.scala:22)
at
$linea2402368d58a4f64b0b63f730078664e25.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command--1:1)
at
$linea2402368d58a4f64b0b63f730078664e25.$read$$iw$$iw$$iw$$iw$$iw.<init>(command--1:43)
at
$linea2402368d58a4f64b0b63f730078664e25.$read$$iw$$iw$$iw$$iw.<init>(command--1:45)
at
$linea2402368d58a4f64b0b63f730078664e25.$read$$iw$$iw$$iw.<init>(command--1:47)
at
$linea2402368d58a4f64b0b63f730078664e25.$read$$iw$$iw.<init>(command--1:49)
at
$linea2402368d58a4f64b0b63f730078664e25.$read$$iw.<init>(command--1:51)
at $linea2402368d58a4f64b0b63f730078664e25.$read.<init>(command--1:53)
at $linea2402368d58a4f64b0b63f730078664e25.$read$.<init>(command--1:57)
at $linea2402368d58a4f64b0b63f730078664e25.$read$.<clinit>(command--1)
at
$linea2402368d58a4f64b0b63f730078664e25.$eval$.$print$lzycompute(<notebook>:7)
at $linea2402368d58a4f64b0b63f730078664e25.$eval$.$print(<notebook>:6)
at $linea2402368d58a4f64b0b63f730078664e25.$eval.$print(<notebook>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:747)
at
scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1020)
at
scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:568)
at
scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:36)
at
scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:116)
at
scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:567)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:594)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:564)
at
com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:223)
at
com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$1(ScalaDriverLocal.scala:237)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at
com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:1564)
at
com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:1489)
at
com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:237)
at
com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$39(DriverLocal.scala:1130)
at com.databricks.unity.EmptyHandle$.runWith(UCSHandle.scala:133)
at
com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$27(DriverLocal.scala:1121)
at
com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at
com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)
at
com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)
at
com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)
at
com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:94)
at
com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472)
at
com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)
at
com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:94)
at
com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$1(DriverLocal.scala:1057)
at
com.databricks.backend.daemon.driver.DriverLocal$.$anonfun$maybeSynchronizeExecution$4(DriverLocal.scala:1517)
at
com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:739)
at
com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$2(DriverWrapper.scala:798)
at scala.util.Try$.apply(Try.scala:213)
at
com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$1(DriverWrapper.scala:790)
at
com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$3(DriverWrapper.scala:830)
at
com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:669)
at
com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:687)
at
com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at
com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)
at
com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)
at
com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)
at
com.databricks.backend.daemon.driver.DriverWrapper.withAttributionContext(DriverWrapper.scala:72)
at
com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472)
at
com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)
at
com.databricks.backend.daemon.driver.DriverWrapper.withAttributionTags(DriverWrapper.scala:72)
at
com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:664)
at
com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:582)
at
com.databricks.backend.daemon.driver.DriverWrapper.recordOperationWithResultTags(DriverWrapper.scala:72)
at
com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:830)
at
com.databricks.backend.daemon.driver.DriverWrapper.executeCommandAndGetError(DriverWrapper.scala:697)
at
com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:742)
at
com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$runInnerLoop$1(DriverWrapper.scala:572)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at
com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at
com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)
at
com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)
at
com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)
at
com.databricks.backend.daemon.driver.DriverWrapper.withAttributionContext(DriverWrapper.scala:72)
at
com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:572)
at
com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:482)
at
com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:290)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at
sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:476)
at
sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:470)
at
sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
at
sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1373)
at sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:73)
at
sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:982)
at
org.apache.iceberg.shaded.org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:149)
at
org.apache.iceberg.shaded.org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at
org.apache.iceberg.shaded.org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:247)
at
org.apache.iceberg.shaded.org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:54)
at
org.apache.iceberg.shaded.org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:299)
at
org.apache.iceberg.shaded.org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:175)
at
org.apache.iceberg.shaded.org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:218)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:717)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:216)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:116)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:188)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
at
org.apache.iceberg.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:123)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:295)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:252)
at org.apache.iceberg.rest.HTTPClient.post(HTTPClient.java:358)
at org.apache.iceberg.rest.RESTClient.post(RESTClient.java:112)
at
org.apache.iceberg.rest.RESTTableOperations.commit(RESTTableOperations.java:152)
at
org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:408)
at
org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at
org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:382)
at
org.apache.iceberg.BaseRewriteManifests.commit(BaseRewriteManifests.java:48)
at
org.apache.iceberg.spark.actions.BaseSnapshotUpdateSparkAction.commit(BaseSnapshotUpdateSparkAction.java:41)
at
org.apache.iceberg.spark.actions.RewriteManifestsSparkAction.replaceManifests(RewriteManifestsSparkAction.java:348)
at
org.apache.iceberg.spark.actions.RewriteManifestsSparkAction.doExecute(RewriteManifestsSparkAction.java:186)
at
org.apache.iceberg.spark.JobGroupUtils.withJobGroupInfo(JobGroupUtils.java:59)
at
org.apache.iceberg.spark.JobGroupUtils.withJobGroupInfo(JobGroupUtils.java:51)
at
org.apache.iceberg.spark.actions.BaseSparkAction.withJobGroupInfo(BaseSparkAction.java:130)
at
org.apache.iceberg.spark.actions.RewriteManifestsSparkAction.execute(RewriteManifestsSparkAction.java:167)
at
wcdp2.maintenance.core.ManifestCompactor$.compact(ManifestCompactor.scala:11)
at
wcdp2.maintenance.core.TableMaintainer$.runCompaction(TableMaintainer.scala:20)
at wcdp2.maintenance.Main$.$anonfun$main$1(Main.scala:23)
at wcdp2.maintenance.Main$.$anonfun$main$1$adapted(Main.scala:22)
at scala.collection.immutable.List.foreach(List.scala:431)
at wcdp2.maintenance.Main$.main(Main.scala:22)
at
$linea2402368d58a4f64b0b63f730078664e25.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command--1:1)
at
$linea2402368d58a4f64b0b63f730078664e25.$read$$iw$$iw$$iw$$iw$$iw.<init>(command--1:43)
at
$linea2402368d58a4f64b0b63f730078664e25.$read$$iw$$iw$$iw$$iw.<init>(command--1:45)
at
$linea2402368d58a4f64b0b63f730078664e25.$read$$iw$$iw$$iw.<init>(command--1:47)
at
$linea2402368d58a4f64b0b63f730078664e25.$read$$iw$$iw.<init>(command--1:49)
at
$linea2402368d58a4f64b0b63f730078664e25.$read$$iw.<init>(command--1:51)
at $linea2402368d58a4f64b0b63f730078664e25.$read.<init>(command--1:53)
at $linea2402368d58a4f64b0b63f730078664e25.$read$.<init>(command--1:57)
at $linea2402368d58a4f64b0b63f730078664e25.$read$.<clinit>(command--1)
at
$linea2402368d58a4f64b0b63f730078664e25.$eval$.$print$lzycompute(<notebook>:7)
at $linea2402368d58a4f64b0b63f730078664e25.$eval$.$print(<notebook>:6)
at $linea2402368d58a4f64b0b63f730078664e25.$eval.$print(<notebook>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:747)
at
scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1020)
at
scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:568)
at
scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:36)
at
scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:116)
at
scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:567)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:594)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:564)
at
com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:223)
at
com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$1(ScalaDriverLocal.scala:237)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at
com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:1564)
at
com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:1489)
at
com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:237)
at
com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$39(DriverLocal.scala:1130)
at com.databricks.unity.EmptyHandle$.runWith(UCSHandle.scala:133)
at
com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$27(DriverLocal.scala:1121)
at
com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at
com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)
at
com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)
at
com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)
at
com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:94)
at
com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472)
at
com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)
at
com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:94)
at
com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$1(DriverLocal.scala:1057)
at
com.databricks.backend.daemon.driver.DriverLocal$.$anonfun$maybeSynchronizeExecution$4(DriverLocal.scala:1517)
at
com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:739)
at
com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$2(DriverWrapper.scala:798)
at scala.util.Try$.apply(Try.scala:213)
at
com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$1(DriverWrapper.scala:790)
at
com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$3(DriverWrapper.scala:830)
at
com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:669)
at
com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:687)
at
com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at
com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)
at
com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)
at
com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)
at
com.databricks.backend.daemon.driver.DriverWrapper.withAttributionContext(DriverWrapper.scala:72)
at
com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472)
at
com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)
at
com.databricks.backend.daemon.driver.DriverWrapper.withAttributionTags(DriverWrapper.scala:72)
at
com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:664)
at
com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:582)
at
com.databricks.backend.daemon.driver.DriverWrapper.recordOperationWithResultTags(DriverWrapper.scala:72)
at
com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:830)
at
com.databricks.backend.daemon.driver.DriverWrapper.executeCommandAndGetError(DriverWrapper.scala:697)
at
com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:742)
at
com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$runInnerLoop$1(DriverWrapper.scala:572)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at
com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
at
com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216)
at
com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424)
at
com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)
at
com.databricks.backend.daemon.driver.DriverWrapper.withAttributionContext(DriverWrapper.scala:72)
at
com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:572)
at
com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:482)
at
com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:290)
at java.lang.Thread.run(Thread.java:750)`
Then any subsequent attempts to access this table will lead to missing files:
`BlobStorageException: Status code 404, "<?xml version="1.0"
encoding="utf-8"?><Error><Code>BlobNotFound</Code><Message>The specified blob
does not exist.
RequestId:e8c4eecc-e01e-003f-65fe-a99772000000
Time:2025-04-10T09:51:02.3477232Z</Message></Error>"
RequestId:e8c4eecc-e01e-003f-65fe-a99772000000
Time:2025-04-10T09:51:02.3477232Z</Message></Error>"
at
com.azure.storage.blob.implementation.util.ModelHelper.mapToBlobStorageException(ModelHelper.java:483)
at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3797)
at reactor.core.publisher.Mono.lambda$onErrorResume$32(Mono.java:3887)
at
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
at
reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)
at
reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192)
at
reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259)
at
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:142)
at
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
at
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
at
reactor.core.publisher.MonoUsing$MonoUsingSubscriber.onNext(MonoUsing.java:231)
at
reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:126)
at
reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224)
at
reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:113)
at
reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:191)
at
reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
at
reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
at
reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:129)
at
reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
at
reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
at
reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:415)
at
reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:439)
at
reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:493)
at
reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:789)
at
reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
com.azure.core.http.netty.implementation.AzureSdkHandler.channelRead(AzureSdkHandler.java:224)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
org.apache.iceberg.azure.shaded.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at
org.apache.iceberg.azure.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at
org.apache.iceberg.azure.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at
org.apache.iceberg.azure.shaded.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
org.apache.iceberg.azure.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)
at
org.apache.iceberg.azure.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338)
at
org.apache.iceberg.azure.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387)
at
org.apache.iceberg.azure.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
at
org.apache.iceberg.azure.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
at
org.apache.iceberg.azure.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
org.apache.iceberg.azure.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at
org.apache.iceberg.azure.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
org.apache.iceberg.azure.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
at
org.apache.iceberg.azure.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at
org.apache.iceberg.azure.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at
org.apache.iceberg.azure.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at
org.apache.iceberg.azure.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at
org.apache.iceberg.azure.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at
org.apache.iceberg.azure.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
at
org.apache.iceberg.azure.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
org.apache.iceberg.azure.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)`
The reason for this seems to be:
https://github.com/apache/iceberg/blob/e5308e8412e817c072e8f08fc5f3186192dadf0f/core/src/main/java/org/apache/iceberg/SnapshotProducer.java#L473
Is there a way to make it not clean up on such failures as it would be
preferable to having to manually restore the blob files
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
