richardstartin commented on issue #7929: URL: https://github.com/apache/pinot/issues/7929#issuecomment-997332098
>If you want to allocate memory in the realtime context, please use the class RealtimeIndexOffheapMemoryManager . It will make sure to allocate memory as per the configuration set by the admin -- memory mapped, or direct, etc. It is also accounted for, and so can be used for provisioning purpose. Before my change `ByteBuffer.allocateDirect` was used, but I'll make a note of this. > Yes, we should not modify general logic for outlier use cases. In this case, we had a production issue, and had to revert the deployment, and spend multiple days trying to reproduce the problem, narrowing down the commit and then identifying a problem within that commit. I apologise for the inconvenience caused by the change, but it's anything but an outlier use case. New user experience has to be considered - if a new user stores text (web scraping, XML, JSON, typically no control over max length) in a raw index, they will find it requires a lot of memory. How do we prevent that new user from having a bad experience at the same time as avoiding finding out about inadvertent regressions in your production environment? Just not making changes would be a good way to avoid regressions in your production environment, but it leaves new users whose workloads are slightly different to yours out in the cold. To avoid a repeat incident, firstly, we need better performance tests so that unconsidered codepaths can't regress without anyone knowing. Secondly, I repeat the suggestion to decouple fixed width and variable width storage, because the implementation has a downside either way so long as it's shared. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
