kishoreg commented on issue #4317: Support variable length Offline Dictionary Indexes for bytes, strings and maps to save on storage URL: https://github.com/apache/incubator-pinot/issues/4317#issuecomment-504133706 > @mcvsubbu Here is the PR for this issue. #4321 > > > Ideally, we should have a VarByte Indexed/Unindexed Reader/Writer with no notion of expansion. Multiple buffer implementations should just be wrappers on top of these things. > > @kishoreg For my own understanding, where do we need Unindexed Reader/Writer? Also, if the entries are variable length, you have to store the length for sure, if not index. And, the storage footprint might be on same lines whether we store index or length field because they're both going to be integers. You are right, we dont need UnIndexedReader/Writer. I was under the impression that we store (length, byte[]) (length, byte[]) in VarByteChunkWriter but it's similar what you have done for VarByteDictionary.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
