coderfender commented on PR #21817: URL: https://github.com/apache/datafusion/pull/21817#issuecomment-4310793279
> I think it would be nice to avoid the `roaring` dependency. > > A plain `BooleanArray` bitmap could work as well? Thanks for the suggestion ! I considered using a plain BooleanArray bitmap, but the challenge is memory usage for sparse key distributions where roaring would be a good fit. If lets say we have -Int max , 0 , +Int max we might be unduly allocating entire u32 range in memory which wouldnt be ideal. Do you have concerns about the roaring dependency specifically (e.g., maintenance, compile time, binary size)? Happy to explore further options -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
