jackjlli commented on PR #12838:
URL: https://github.com/apache/pinot/pull/12838#issuecomment-2046224113
I did test out the cases when forward index is disabled for the realtime
table. And here is the exception messages:
```
2024/04/09 16:31:42.220 INFO [FixedByteSVMutableForwardIndex]
[HelixTaskExecutor-message_handle_thread_55] Allocating 200 bytes for:
airlineStats__7__1__20240409T2331Z:$ts$DAY.sv.unsorted.fwd
2024/04/09 16:31:42.220 INFO [FixedByteSVMutableForwardIndex]
[HelixTaskExecutor-message_handle_thread_55] Allocating 512 bytes for:
airlineStats__7__1__20240409T2331Z:$ts$MONTH.dict
2024/04/09 16:31:42.220 INFO [FixedByteSVMutableForwardIndex]
[HelixTaskExecutor-message_handle_thread_55] Allocating 200 bytes for:
airlineStats__7__1__20240409T2331Z:$ts$MONTH.sv.unsorted.fwd
2024/04/09 16:31:42.220 INFO [FixedByteSVMutableForwardIndex]
[HelixTaskExecutor-message_handle_thread_55] Allocating 512 bytes for:
airlineStats__7__1__20240409T2331Z:$ts$WEEK.dict
2024/04/09 16:31:42.220 INFO [FixedByteSVMutableForwardIndex]
[HelixTaskExecutor-message_handle_thread_55] Allocating 200 bytes for:
airlineStats__7__1__20240409T2331Z:$ts$WEEK.sv.unsorted.fwd
2024/04/09 16:31:42.220 INFO [FixedByteSVMutableForwardIndex]
[HelixTaskExecutor-message_handle_thread_55] Allocating 256 bytes for:
airlineStats__7__1__20240409T2331Z:ActualElapsedTime.dict
2024/04/09 16:31:42.220 INFO [FixedByteSVMutableForwardIndex]
[HelixTaskExecutor-message_handle_thread_55] Allocating 200 bytes for:
airlineStats__7__1__20240409T2331Z:ActualElapsedTime.sv.unsorted.fwd
2024/04/09 16:31:42.221 INFO [FixedByteSVMutableForwardIndex]
[HelixTaskExecutor-message_handle_thread_55] Allocating 256 bytes for:
airlineStats__7__1__20240409T2331Z:AirTime.dict
...
2024/04/09 16:31:42.232 WARN
[RealtimeSegmentDataManager_airlineStats__6__1__20240409T2331Z]
[HelixTaskExecutor-message_handle_thread_54] Scheduling task to call controller
to mark the segment as OFFLINE in Ideal State due to initialization error:
'Forward index is required'
2024/04/09 16:31:42.232 ERROR [216_7050 - SegmentOnlineOfflineStateModel]
[HelixTaskExecutor-message_handle_thread_55] Caught exception in state
transition OFFLINE -> CONSUMING for table: airlineStats_REALTIME, segment:
airlineStats__7__1__20240409T2331Z
java.lang.IllegalArgumentException: Forward index is required
at
org.apache.pinot.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-c7f5887d55981764646d873b836eaed384ed4339]
at
org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl$IndexContainer.<init>(MutableSegmentImpl.java:1304)
~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-c7f5887d55981764646d873b836eaed384ed4339]
at
org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:364)
~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-c7f5887d55981764646d873b836eaed384ed4339]
at
org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.<init>(RealtimeSegmentDataManager.java:1529)
~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-c7f5887d55981764646d873b836eaed384ed4339]
at
org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:462)
~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-c7f5887d55981764646d873b836eaed384ed4339]
at
org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:241)
~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-c7f5887d55981764646d873b836eaed384ed4339]
at
org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline(SegmentOnlineOfflineStateModelFactory.java:91)
~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-c7f5887d55981764646d873b836eaed384ed4339]
at
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350)
~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-c7f5887d55981764646d873b836eaed384ed4339]
at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278)
~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-c7f5887d55981764646d873b836eaed384ed4339]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97)
~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-c7f5887d55981764646d873b836eaed384ed4339]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49)
~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-c7f5887d55981764646d873b836eaed384ed4339]
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
~[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
```
updated the fieldConfig to the realtime table config:
```
@@ -55,6 +57,19 @@
"MONTH"
]
}
+ },
+ {
+ "name": "AirTime",
+ "encodingType": "DICTIONARY",
+ "indexType": "INVERTED",
+ "indexTypes": [
+ "INVERTED"
+ ],
+ "indexes": null,
+ "properties": {
+ "forwardIndexDisabled": "true"
+ },
+ "tierOverwrites": null
}
],
"metadata": {
```
So I think it still makes sense to fail fast on table config update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
