egalpin commented on PR #10704: URL: https://github.com/apache/pinot/pull/10704#issuecomment-1533370590
Ya that makes sense, otherwise it would literally be changing the semantics of which docs should have been persisted in the first place. I do want to allow for support of adding a brand new column to the schema + comparisonColumns list though, which I agree should work. When reloading sealed segments, is processing done in global order of `docId`? Or are segments loaded in parallel? If they're loaded strictly in global order of docId, I feel we might not need any comparison at all; effectively, any doc with a higher docId was persisted _after_ a doc with a lower docId and therefore whatever the upsert comparison semantics were at the time when the doc was persisted _must_ have been truthy (otherwise the doc would have been rejected). Therefore we could ensure to maintain whatever the prior upsert semantics were, then use the new semantics moving forward, by forcing `compareTo` to return `1` for every doc read from a sealed segment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
