mcvsubbu commented on issue #7929: URL: https://github.com/apache/pinot/issues/7929#issuecomment-997323678
> For some context about the change and what this will go back to, if you happen to have an outlier record in a set of JSON records, of say 1MB (which isn’t that large) compared to 10KB on average, you need ~1GB for the buffer, then ~2GB for the compression buffer. Saving 2GB per raw index build is a really nice improvement, if that’s how you’re using Pinot. Perhaps the problem is sharing the logic, which isn’t particularly complicated, between fixed and variable length data sources? Yes, we should not modify general logic for outlier use cases. In this case, we had a production issue, and had to revert the deployment, and spend multiple days trying to reproduce the problem, narrowing down the commit and then identifying a problem within that commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
