lksvenoy-r7 commented on issue #9082: URL: https://github.com/apache/pinot/issues/9082#issuecomment-1192331732
Thanks for the well thought out reply @Jackie-Jiang. I haven't spent much time thinking about the implications of this feature, and these observations are great. 1. You're right that this could lead to much higher storage cost, perhaps it would be possible to have some type of limit option for creating the permutations? I'm not sure this makes sense, as the behaviour could be unpredictable. There would be no way of knowing which types of multi-valued column values have been indexed, so it would have to be discussed whether this is an acceptable trade-off or not. 2. What exactly do you mean? Are you saying storing all permutations, or simply one index split per value? If it is the latter as I had written up in my issue, then that makes sense. However, if it's all possible permutations being indexed, then this would probably not be feasible. We would then suffer exponential storage explosion due to all the possible permutations. This is why I am arguing that an acceptable trade-off would be to allow a lookup on simple one value 3. I hadn't considered the implications on the star-node. This seems like one of the most challenging problems to solve, and it makes it seem like perhaps the star node is not compatible at that level. We will need to think about this one more.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
