npawar opened a new issue #6308:
URL: https://github.com/apache/incubator-pinot/issues/6308
Just in the last day, we had 2 reports of consumption stopping completely
for a partition, because of some error occurrences in consumption.
Example 1:
User saw this error, and reported that consumption is totally stopped for
that partition.
```
ERROR [LLRealtimeSegmentDataManager_spanEventView__0__15__20201201T0448Z]
[spanEventView__0__15__20201201T0448Z] Could not build segment
java.lang.IllegalStateException: Cannot create output dir:
/var/pinot/server/data/index/spanEventView_REALTIME/_tmp/tmp-spanEventView__0__15__20201201T0448Z-160688315943
```
This is coming from LLRealtimeSegmentDataManager#buildSegmentInternal(). As
a result of this, we reach this in LLRealtimeSegmentDataManager
```
// We could not build the segment. Go into error state.
_state = State.ERROR;
```
Problem: After that, the consumer thread exits. I believe an assumption is
made here that some other replica will have built the segment, and this errored
replica can simply download the segment. **But this assumption fails if user
has only 1 replica.**
Lack of detection mechanism: Because the ideal state and external view
match, the Cluster Manager continues to report GOOD. Users find out of the
problem because they notice data lag in their application. Most users who are
just starting out have not set up metrics monitoring, so asking them to look
for LLC_PARTITION_CONSUMING metric is also not an option.
Lack of easy recovery mechanism: The validation manager does nothing to fix
this, because the ideal state is still CONSUMING and segment metadata is still
IN_PROGRESS. Only way to recover is restarting that server.
Example 2:
Again, user reported that consumption is totally stopped on the partition,
and reported this error.
```
Exception while executing a state transition task
realtimesolines__234__4__20201127T1841Z
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_77]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_77]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_77]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]
at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404)
~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb9
6cb36efb3e4dbceb10461e73c0d6465e81493]
at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331)
[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSH
OT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97)
[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49)
[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_77]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:693) ~[?:1.8.0_77]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
~[?:1.8.0_77]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_77]
at
org.apache.pinot.core.segment.memory.PinotByteBuffer.allocateDirect(PinotByteBuffer.java:38)
~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4dbceb
10461e73c0d6465e81493]
at
org.apache.pinot.core.segment.memory.PinotDataBuffer.allocateDirect(PinotDataBuffer.java:116)
~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4dbce
b10461e73c0d6465e81493]
at
org.apache.pinot.core.io.writer.impl.DirectMemoryManager.allocateInternal(DirectMemoryManager.java:53)
~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36e
fb3e4dbceb10461e73c0d6465e81493]
at
org.apache.pinot.core.io.readerwriter.RealtimeIndexOffHeapMemoryManager.allocate(RealtimeIndexOffHeapMemoryManager.java:79)
~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6
.0-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
at
org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.addBuffer(FixedByteSVMutableForwardIndex.java:208)
~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0
.6.0-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
at
org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.<init>(FixedByteSVMutableForwardIndex.java:77)
~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0
-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
at
org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:294)
~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4
dbceb10461e73c0d6465e81493]
at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.<init>(LLRealtimeSegmentDataManager.java:1245)
~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-S
NAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
at
org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:312)
~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSH
OT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
at
org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:133)
~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAP
SHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
at
org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:164)
~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
at
org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline(SegmentOnlineOfflineStateModelFactory.java:8
8)
~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
... 12 more
```
This appears to have happened when the state transition for OFFLINE to
CONSUMING was being processed (during memory allocation for beginning
consumption of a new segment. The consumption loop hasn't been reached). The
state transition method `onBecomeOnlineFromOffline` threw the exception, and
the segment must've gone into ERROR state in the external view. Ideal state
will still be CONSUMING, and segment metadata will be IN_PROGRESS.
Problem: Similar to above case, the ideal state and segment metadata
indicate that nothing is wrong with the system. But consumption is not
happening, the consumer thread never even started.
Lack of detection mechanism: This should've been flagged on the Cluster
Manager as BAD, as there is a mismatch between IS and EV.
Lack of recovery mechanism: Again, the validation manager won't fix this,
because it only looks at Ideal State, and according to the IS everything looks
fine. Only way to recover will be restart of the server.
We need
1. A way to restart consumption automatically in both cases - Possibly make
Validation Manager handle both these cases.
2. A way to track state of consumers for every CONSUMING segment - an API on
the controller would be ideal
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
