somandal commented on issue #10658: URL: https://github.com/apache/pinot/issues/10658#issuecomment-1520941010
Thanks @vvivekiyer I'll take a look at this in more detail soon. Just wanted to bring up what we discussed offline: > Do not introduce an ARRAY datatype. But equate/evolve our support of MV columns to achieve all the support that ARRAYs have. I do have some concerns with this. MV today doesn't have a very clear guideline on whether it's really like an ARRAY or a SET or something else. For example (as we discussed offline), today when the `ForwardIndexHandler` disables the forward index for an MV column and wants to reconstruct it later on, the following guarantees cannot be met: - The order of the elements in each MV row can be reordered (i.e. ordering is not preserved on re-enabling the forward index). This breaks array semantics - Today when an MV row has duplicate entries, on forward index reconstruction the duplicates are lost as we don't store frequency information. This breaks array semantics. Though we don't reorder MV rows (as far as I can tell) anywhere else in the code, these semantics aren't ingrained into Pinot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
