mcvsubbu commented on issue #6679: URL: https://github.com/apache/incubator-pinot/issues/6679#issuecomment-876584254
One of the problems faced is that kafka topic (whether new topic on the same table, or some unerlying kafka change) causes offsets to go lower. So, the option to re-add consuming segment should be to pick the latest or earliest available offset (as per table config) even if it is lower than the previous segment. So, maybe an option like `ignoreOffsetConsistency` ? Operator can provide an offset, but it is hard to do that on a per partition basis. They will need to enter a command for each partition, maybe. I would say provide a way to create consuming segments for _all_ partitions with one command. Optionally, the operator can specify partition numbers, and we can re-create consuming segments for those partitions. Depending on the boolean to ignore the consistency, we can either return an error or force create a consuming segment. If we are force creating, just bump up the sequence number far enough out (say, +100) so that we know by looking at things that something changed (when they call for support -- this is experimental, and there is no saying what else went wrong). Let us be really careful here, and offer minimum features on this. Such scenarios have not been tested, so this is not a beginner task to go and validate possible race conditions via manual testing. Good thing is we know that the automatic recovery ignores this condition. I would like it to stay that way. In order to add a consuming segment, you need to do the following: (1) Add segment metadata (2) Update the idealstate. Not sure what happens when automatic correction comes around between (1) and (2). We will need to play that out manually and see. A safe method of doing this is perhaps to send out a helix broadcast to all controllers when the API is received, and let the controller that is master for that table handle it, perhaps while locking out the realtime fixer job. it will also be useful to NOT allow deletion of consuming segments unless the admin explicitly says so in the segment deletion API. I suggest adding a `includeConsumingSegments` option there. All this makes it clear, that the admins have the knobs to turn around for automatic fixes, but when they start doing things manually, they are on a thin line. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
