tibrewalpratik17 commented on issue #13683: URL: https://github.com/apache/pinot/issues/13683#issuecomment-2250784126
> Because technically that would be the right thing to fix here right? (or are we saying that there's no way to improve upon https://github.com/apache/pinot/pull/13256, and this is something we will have to deal with for partial upsert tables going forward Yeah for data-consistency guarantees, we need to release the semaphore post the segment has been committed completely (immutable segment replaced mutable segment). One thing which i was thinking is that in the new consuming segment thread during the state change `OFFLINE -> CONSUMING`, we are waiting for the lock release. Maybe if we can move that operation to another async thread and change the segment state irrespective then that might help. But that is more of a fundamental change which will affect all types of tables (upsert / non-upsert ones) so we can leave that to another discussion. For our case specifically, we saw the segment build time is taking ~7 mins likely because the number of columns are 300+. We can use this #13686 as a fix for this particular use-case and then discuss about the general fix. Wdyt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
