kishoreg commented on issue #4317: Support variable length Offline Dictionary Indexes for bytes, strings and maps to save on storage URL: https://github.com/apache/incubator-pinot/issues/4317#issuecomment-503277146 Take a look at the code within MutableOffHeapByteArrayStore.Buffer. That's exactly along the lines of what we need for a Var length Dictionary. But that's tightly coupled with MutableOffHeapByteArrayStore (e.g. it has the startIndex, which should have been managed within MutableOffHeapByteArrayStore). There are few more things such as in real-time we don't know the number of entries upfront but in case of the immutable dictionary, we know the exact number of entries, size of each entry and total size. There might be some additional optimizations we can do based on these things. This is a classic case for cuckoo hashing as well. One possibility is to extract out the MutableOffHeapByteArrayStore.Buffer inner class as VarByteReaderWriter. Let's get this feature in without touching any of the existing functionalities. I can do the cleanup in another PR.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
