arkadius opened a new issue, #11197:
URL: https://github.com/apache/iceberg/issues/11197
### Query engine
Flink 1.19.1
### Question
Hi, I'm using Flink with Iceberg 1.6.1 (I also tried the current snapshot).
While inserting the `Row` into a Iceberg table I'm getting an error:
```
com.esotericsoftware.kryo.KryoException: java.lang.ArrayStoreException:
org.apache.iceberg.GenericDataFile
Serialization trace:
dataFiles (org.apache.iceberg.io.WriteResult)
at
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
~[flink-dist-1.19.1.jar:1.19.1]
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
~[flink-dist-1.19.1.jar:1.19.1]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:392)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:206)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:44)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:53)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.runtime.io.network.api.serialization.NonSpanningWrapper.readInto(NonSpanningWrapper.java:337)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.readNonSpanningRecord(SpillingAdaptiveSpanningRecordDeserializer.java:128)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.readNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:103)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:93)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:103)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:579)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
~[flink-dist-1.19.1.jar:1.19.1]
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751)
[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
[flink-dist-1.19.1.jar:1.19.1]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.ArrayStoreException: org.apache.iceberg.GenericDataFile
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:378)
~[flink-dist-1.19.1.jar:1.19.1]
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:289)
~[flink-dist-1.19.1.jar:1.19.1]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
~[flink-dist-1.19.1.jar:1.19.1]
at
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
~[flink-dist-1.19.1.jar:1.19.1]
... 21 more
```
Unfortunately, it is non-deterministic. By non-deterministic I mean, that
all the time I insert the same `Row` but sometimes this error is thrown and
sometimes not. When it isn't thrown, records appear in the target table. My job
uses Flink's table-api. Here is the catalog configuration:
```
CREATE CATALOG nessie_catalog WITH (
'type'='iceberg',
'catalog-impl'='org.apache.iceberg.nessie.NessieCatalog',
'io-impl'='org.apache.iceberg.aws.s3.S3FileIO',
'uri'='http://nessie:19120/api/v2',
'warehouse' = 's3://warehouse',
's3.endpoint'='http://minio:9000'
);
```
I see, that some changes that are currently being made to the sinks.
(#10179). Do you think that they could help with my problem? Do you have any
idea what introduces this non-determinism?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
