tibrewalpratik17 commented on issue #13683: URL: https://github.com/apache/pinot/issues/13683#issuecomment-2252429945
> could we instead use a separate semaphore for blocking partial upsert table consumption in PartitionConsumer#run (or there might be some other better place)? The idea is that to ensure consistency for Partial Upserts, it's enough to make sure the consumption doesn't start before the previous segment is completely destroyed. To do that, using the current semaphore is a bit of an overkill since it straight up blocks the creation of the segment data manager. Instead, all we want to block in that case is the consumption, which could be done in a more fine-grained manner by doing it in PartitionConsumer. I agree that this approach makes more sense, but I'm a bit concerned about potential repercussions. During the state transition from `OFFLINE -> CONSUMING`, we create the segment-data-manager and then register it. If we register it without starting consumption, there might be side effects that I can't fully identify yet. I'm open to exploring that route as well. However, since it might involve significant changes (considering the fundamental nature of Helix transitions), could we quickly patch the configuration proposed in #13686 to unblock our production issues for now? Allowing flexibility to fine-tune the segment-expiration configuration wouldn't have any major side effects and would be beneficial in other scenarios where build times are high. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
