harold-kfuse opened a new issue #7603:
URL: https://github.com/apache/pinot/issues/7603
I'm using Pinot 0.8.0 deployed in GKE (using the official published 0.8.0
image in dockerhub: apachepinot/pinot:0.8.0). I have a realtime table that
consumes from Kafka. I noticed that it stopped consuming from Kafka. While
looking at the logs, I saw the following exception:
```
Exception in thread "XXXX__0__52__20211019T2128Z" java.lang.InternalError: a
fault occurred in a recent unsafe memory access operation in compiled Java code
at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
at
java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:172)
at
java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538)
at java.base/java.lang.StringBuilder.append(StringBuilder.java:174)
at java.base/java.util.Arrays.toString(Arrays.java:5116)
at
org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer.standardize(DataTypeTransformer.java:170)
at
org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer.transform(DataTypeTransformer.java:90)
at
org.apache.pinot.segment.local.recordtransformer.CompositeTransformer.transform(CompositeTransformer.java:82)
at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:496)
at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:417)
at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:560)
at java.base/java.lang.Thread.run(Thread.java:829)
```
This is my table configuration [redacted some information]:
```
{
"REALTIME": {
"tableName": "XXXX_REALTIME",
"tableType": "REALTIME",
"segmentsConfig": {
"schemaName": "XXXX",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "14",
"timeColumnName": "ts",
"replication": "1",
"replicasPerPartition": "1"
},
"tenants": {
"broker": "DefaultTenant",
"server": "realtime"
},
"tableIndexConfig": {
"invertedIndexColumns": [],
"noDictionaryColumns": [],
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.topic.name": "pinot_topic",
"stream.kafka.broker.list": "kafka:9092",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest",
"stream.kafka.consumer.factory.class.name":
"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"realtime.segment.flush.threshold.time": "6h",
"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.segment.size": "100M",
"stream.kafka.decoder.class.name": "XXXXX",
},
"onHeapDictionaryColumns": [],
"varLengthDictionaryColumns": [],
"enableDefaultStarTree": false,
"enableDynamicStarTreeCreation": false,
"aggregateMetrics": false,
"nullHandlingEnabled": false,
"rangeIndexColumns": [],
"autoGeneratedInvertedIndex": false,
"createInvertedIndexDuringSegmentGeneration": false,
"bloomFilterColumns": [],
"loadMode": "MMAP"
},
"metadata": {},
"isDimTable": false
}
}
```
We do have our own decoder plugin that reads binary protobuf from kafka,
deserialize it, then populates the GenericRow.
I saw this issue:
https://github.com/apache/pinot/issues/7476
Stack trace is not exactly the same, so I'm not sure if this is the same
issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
