chrajeshbabu opened a new issue, #17298: URL: https://github.com/apache/pinot/issues/17298
Currently, Apache Pinot supports star-tree indexes for pre-aggregation, but sometimes creating star-trees for wide-column tables is not effective due to excessive disk space consumption. To address this, introduce support for Materialised Views (MV) in Pinot: **MV Creation:** Allow materialised view tables to be created externally (e.g., in Hive) and ingested into Pinot as MV tables. **Metadata Mapping:** Maintain metadata linking MV tables to the main table, including the original query used for MV creation. **Query Rewrite:** Integrate Calcite-based query rewriting to automatically select the most cost-effective MV table when serving queries. **Further Optimisation:** Enable star-tree indexes on MV tables for frequently accessed aggregations. Since MV tables are smaller, star-trees will consume less space, improving performance. **Goal:** Achieve multi-level aggregation and efficient real-time analytics for wide-column tables without prohibitive storage costs. Pros - Significant performance boost for ad-hoc queries and complex aggregations. - Reduced disk space usage compared to star-trees on wide-column tables. - Flexible design leveraging existing MV creation tools like Hive. Cons - Users must refresh MV tables when underlying data changes. - Best suited for offline tables rather than real-time ingestion scenarios. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
