kishoreg commented on issue #4317: Support variable length Offline Dictionary Indexes for bytes, strings and maps to save on storage URL: https://github.com/apache/incubator-pinot/issues/4317#issuecomment-502851088 From @buchireddy @mcvsubbu I did look at MutableOffheapByteArrayStore and took some concepts from it while implementing VarLengthBytesValueReaderWriter too. However, I haven't used the former fully because of the following reasons: In order to support a seamless transition from fixed length to variable length offline dictionary, we need a special magic header to be added at the beginning, which needs some changes to MutableOffheapByteArrayStore. MutableOffheapByteArrayStore is handling the buffer expansions, multiple buffers logic because it needs to be mutable but the immutable one doesn't need all those. So, it's a trade-off b/w simple code vs avoiding the code duplication (though we anyways need a new class for variable length dictionary class implementation). I'm open for debate and for further brainstorming. P.S: This is my first time going through Pinot code so please let me know if I'm missing any aspects here.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
