Jackie-Jiang commented on code in PR #11345:
URL: https://github.com/apache/pinot/pull/11345#discussion_r1294233651
##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/FreshnessBasedConsumptionStatusChecker.java:
##########
@@ -76,15 +90,36 @@ protected boolean isSegmentCaughtUp(String segmentName,
LLRealtimeSegmentDataMan
StreamPartitionMsgOffset currentOffset =
rtSegmentDataManager.getCurrentOffset();
StreamPartitionMsgOffset latestStreamOffset =
rtSegmentDataManager.fetchLatestStreamOffset(5000);
if (isOffsetCaughtUp(currentOffset, latestStreamOffset)) {
- _logger.info("Segment {} with freshness {}ms has not caught up within
min freshness {}."
+ _logger.info("Segment {} with freshness {}ms has not caught up within
min freshness {}. "
Review Comment:
Not introduced in this PR, but does `isOffsetCaughtUp()` never works for
Kafka?
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IdleTimer.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.core.data.manager.realtime;
+
+/**
+ * The IdleTimer is responsible for keeping track of 2 different idle times:
+ * - The stream idle time which resets every time we remake the stream
consumer.
Review Comment:
I don't fully follow the difference between these 2 idle time. Currently
seems we always update and reset them together
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IdleTimer.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.core.data.manager.realtime;
+
+/**
+ * The IdleTimer is responsible for keeping track of 2 different idle times:
+ * - The stream idle time which resets every time we remake the stream
consumer.
+ * This depends on the user configured "idle.timeout.millis" stream config.
+ * - the total idle time which only resets when we consume something.
+ *
+ * This is not a running timer. It only advances as long as we keep calling
"markIdle".
+ * This also makes it sightly inaccurate as we don't start counting idle time
until
+ * we've been idle for the first time. This is fine for our use case as we are
using this
+ * in a fast moving consumeLoop with iterations taking on the order of
milliseconds to seconds.
+ */
+public class IdleTimer {
+
+ private long _streamIdleStartTimeMs = 0;
Review Comment:
If we are going to access them from a different thread, need to make them
thread safe (either `volatile` or add `synchronized`)
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/IdleTimer.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.core.data.manager.realtime;
+
+/**
+ * The IdleTimer is responsible for keeping track of 2 different idle times:
+ * - The stream idle time which resets every time we remake the stream
consumer.
+ * This depends on the user configured "idle.timeout.millis" stream config.
+ * - the total idle time which only resets when we consume something.
+ *
+ * This is not a running timer. It only advances as long as we keep calling
"markIdle".
Review Comment:
Why not just tracking the start time, and ask for `nowMs` when reading the
idle time? If the start time is 0, return 0 then
##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/FreshnessBasedConsumptionStatusChecker.java:
##########
@@ -76,15 +90,36 @@ protected boolean isSegmentCaughtUp(String segmentName,
LLRealtimeSegmentDataMan
StreamPartitionMsgOffset currentOffset =
rtSegmentDataManager.getCurrentOffset();
StreamPartitionMsgOffset latestStreamOffset =
rtSegmentDataManager.fetchLatestStreamOffset(5000);
if (isOffsetCaughtUp(currentOffset, latestStreamOffset)) {
- _logger.info("Segment {} with freshness {}ms has not caught up within
min freshness {}."
+ _logger.info("Segment {} with freshness {}ms has not caught up within
min freshness {}. "
+ "But the current ingested offset is equal to the latest
available offset {}.", segmentName, freshnessMs,
_minFreshnessMs, currentOffset);
return true;
}
+ // If the earliest available kafka offset is later than the current
offset, then for the purposes of
+ // this check, we are caught up. This can happen if the segment has not
consumed data for some time,
+ // and that data has since been retentioned.
+ StreamPartitionMsgOffset earliestStreamOffset =
rtSegmentDataManager.fetchEarliestStreamOffset(5000);
Review Comment:
Do we want to count this as pass? If the consumer works as expected, it
should be able to skip to the next available offset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
