Hi all, I’m aligned with the overall direction here. Supporting BQMS and other native Iceberg catalogs via federation seems useful, and the modularization discussion makes sense to me as well.
One question I wonder if we should make more explicit as this grows: *is federation in Polaris aiming for a common contract over native Iceberg catalogs*, or *primarily a set of backend-specific adapters behind one REST surface*? I think that answer affects a few of the design choices being discussed here: - how strong the common connection/interface should be - whether connector auth/capability constraints should be explicit at the boundary - how much mismatch is expected to be caught at create-time vs later in runtime - how much backend-specific behavior we are comfortable exposing through the common surface My bias would be: keep modularization simple for now, but make connector constraints as explicit as possible at the boundary. Otherwise it feels too easy for the public contract to become broader than what a given backend implementation actually supports. Thanks, -ej On Tue, Mar 24, 2026 at 8:24 PM Joy <[email protected]> wrote: > Hello all, > > I have opened a PR for BigQuery Metastore support, > https://github.com/apache/polaris/pull/4050 > It also adds a BIGQUERY connection type to the REST API. Looking forward to > your feedback. > > Regards, > Joy > > On Fri, Mar 20, 2026 at 8:35 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi Joy, > > > > Thanks - your plan looks good to me! Looking forward to a PR :) > > > > Cheers, > > Dmitri. > > > > On Fri, Mar 20, 2026 at 6:58 AM Joy <[email protected]> wrote: > > > > > Hello all, > > > > > > Thank you so much for all the feedback and guidance. > > > > > > I am trying to summarize what I understood from the thread and some > Slack > > > conversations with JB and Prashant. > > > > > > *On how to structure the code:* > > > > > > JB suggested separate modules per catalog, like > > > *extensions/federation/bqms*, > > > *extensions/federation/glue*. Each module has its own factory class and > > > manages its own dependencies, similar to how Hive federation works > today > > if > > > I understand correctly. > > > > > > Prashant suggested a hybrid, one generic factory that works with > > Iceberg's > > > *BaseMetastoreCatalog* interface. Separate modules would only bring in > > > dependencies, for example, iceberg-bigquery, no custom factory code > would > > > be needed per catalog. > > > > > > *On pluggability:* > > > > > > Dmitri prefers keeping things simple with current Quarkus module > > selection. > > > > > > Romain raised a good point about making it easier for end users who > > consume > > > Polaris as a Docker image. > > > > > > But my understanding is limited here. > > > > > > *My plan:* > > > > > > I'd like to start with a simple approach since most people seem okay > with > > > that. I can try an initial implementation following JB's pattern i.e. > > > separate module for BQMS or Prashant's pattern i.e. generic factory. > > > > > > I'd be okay if the first attempt doesn't get accepted, it may help move > > the > > > discussion forward with some code to look at. > > > > > > Please let me know if I've misunderstood anything or if there's a > > preferred > > > direction. > > > > > > Regards, > > > Joy > > > > > > On Thu, 19 Mar, 2026, 4:52 am Romain Manni-Bucau, < > [email protected] > > > > > > wrote: > > > > > > > Hi, > > > > > > > > As an end users I have a small feedback on that topic: > > > > > > > > 1. it is unlikely we rebuild polaris to get federation (we'd federate > > > > poalris to make it clear) - with hive catalog for now > > > > 2. hive integration with polaris is not fully aligned with spark > which > > > has > > > > shihms so there is already this classloader or alternative need > > > > > > > > > > > > what can be envisionned if you do not want to go with classloaders is > > > real > > > > pluggability at *runtime* (like any real CDI application) or worse > case > > > > using a ServiceLoader (even if weird in a CDI stack since the SPI is > > CDI > > > > impl itself). > > > > > > > > so from my small window a valid option is to make polaris runtime > > > friendly > > > > (vs build friendly which is not a solution for a product) and > > optionally > > > > enable classloading (just needs to be a tree like > > > > > > > > > > > > > > https://github.com/Talend/component-runtime/blob/master/container/container-core/src/main/java/org/talend/sdk/component/container/ContainerManager.java > > > > ). > > > > the classloader support can even be made in a runtime extension which > > > does > > > > load a configuration to load other extension so you get the best of > > both > > > > worlds, but point is you cant request users to *build* for a webapp > > > > ultimately delivered as a docker image - I know apache is just about > > > > sources but nobody consumes it this way (except vendors when they > fork > > > but > > > > hopefully it is a few end users). > > > > > > > > hope it makes sense > > > > > > > > Romain Manni-Bucau > > > > @rmannibucau <https://x.com/rmannibucau> | .NET Blog > > > > <https://dotnetbirdie.github.io/> | Blog < > > https://rmannibucau.github.io/ > > > > > > > > | Old > > > > Blog <http://rmannibucau.wordpress.com> | Github > > > > <https://github.com/rmannibucau> | LinkedIn > > > > <https://www.linkedin.com/in/rmannibucau> | Book > > > > < > > > > > > > > > > https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064 > > > > > > > > > Javaccino founder (Java/.NET service - contact via linkedin) > > > > > > > > > > > > Le mer. 18 mars 2026 à 23:18, Dmitri Bourlatchkov <[email protected]> > a > > > > écrit : > > > > > > > > > Hi All, > > > > > > > > > > Good point about modeling federation implementations similarly to > > > > pluggable > > > > > persistence implementations. > > > > > > > > > > I'm not sure about meddling with ClassLoaders in Polaris. I'd > prefer > > to > > > > > leverage the existing Quarkus-based integration of multiple modules > > > into > > > > a > > > > > single runtime env. (current state). > > > > > > > > > > Downstream project will have the ability to elect which modules to > > use > > > > > at build time. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Wed, Mar 18, 2026 at 2:42 AM Jean-Baptiste Onofré < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > "Classloader" is a fair and valid point. I have shared this > > > perspective > > > > > in > > > > > > the past and discussed it with Nandor as well: I believe > federation > > > > > should > > > > > > utilize a "plugin" mechanism, similar to how persistence is > > handled. > > > > > > > > > > > > By adopting this approach, each plugin (such as HMS) is in its > own > > > > > module, > > > > > > and would be responsible for managing its own dependencies and > > > shading. > > > > > > > > > > > > Regards, > > > > > > JB > > > > > > > > > > > > On Wed, Mar 18, 2026 at 12:37 AM Yufei Gu <[email protected]> > > > > wrote: > > > > > > > > > > > > > Federation seems like the strongest candidate given the > > dependency > > > > > > surface. > > > > > > > For example, Trino-style classloader isolation could help when > > > > > conflicts > > > > > > > arise (e.g., Hadoop or Guava across different federation > > > > > > implementations). > > > > > > > That said, I’d suggest introducing this only once we actually > hit > > > > such > > > > > > > issues. Until then, keeping things simple with proper > > > modularization > > > > > may > > > > > > be > > > > > > > the better tradeoff. WDTY? > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2026 at 2:00 PM Madhan Neethiraj < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > we need to be careful about is proper modularization to > > control > > > > > > > > dependency scope growth. > > > > > > > > > > > > > > > > I share this concern; it is applicable for any extension like > > > > > > > authorizers. > > > > > > > > Having a built-in support for isolation of extension specific > > > > > > > dependencies > > > > > > > > will significantly reduce/eliminate complexities in handling > > > > > potential > > > > > > > > library conflicts and avoid impact to Polaris core due to > > > > > dependencies > > > > > > > > dragged by extensions. Trino's plugin model seems to be a > good > > > > > > reference > > > > > > > - > > > > > > > > 1) and 2). > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Madhan > > > > > > > > > > > > > > > > 1) https://trino.io/docs/current/installation/plugins.html > > > > > > > > 3) https://trino.io/docs/current/develop/spi-overview.html > > > > > > > > > > > > > > > > > > > > > > > > On 3/17/26, 1:30 PM, "Prashant Singh via dev" < > > > > > [email protected] > > > > > > > > <mailto:[email protected]>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > +1 to this. I would also recommend thinking about this from > an > > > > > > interface > > > > > > > > POV to have BaseMetastoreCatalog (from Apache Iceberg) as a > way > > > to > > > > > > > > integrate, with a defined connection object that can feed the > > > > > > connection > > > > > > > to > > > > > > > > BQ or Glue (perhaps a standard key-value prep) > > > > > > > > and the dependencies can be added accordingly. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Prashant Singh > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2026 at 12:08 PM Dmitri Bourlatchkov < > > > > > [email protected] > > > > > > > > <mailto:[email protected]>> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Joy, > > > > > > > > > > > > > > > > > > I suppose Polaris will be the REST API + AuthZ layer on top > > of > > > > > > > > > BigQueryMetastoreCatalog, right? How do you envision this? > > > > > > > > > > > > > > > > > > In general, I think it's going to be a useful contribution. > > > > > > > > > > > > > > > > > > From my POV, the only technical issue we need to be careful > > > about > > > > > is > > > > > > > > > proper modularization to control dependency scope growth. I > > > > imagine > > > > > > > some > > > > > > > > > downstream projects may want to opt in/out of > > > > > > BigQueryMetastoreCatalog > > > > > > > > and > > > > > > > > > its transitive dependencies at their own discretion. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > On 2026/03/17 18:19:34 Joy wrote: > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > I'm Joy. We use BigQueryMetastoreCatalog for Iceberg on > GCP > > > and > > > > > > want > > > > > > > to > > > > > > > > > use > > > > > > > > > > Polaris as a catalog gateway. I see federation currently > > > > supports > > > > > > > REST > > > > > > > > > and > > > > > > > > > > HMS. > > > > > > > > > > > > > > > > > > > > Would there be interest in supporting other native > Iceberg > > > > > catalogs > > > > > > > > like > > > > > > > > > > BQMS or Glue? I've been working on a prototype for BQMS > > > > following > > > > > > the > > > > > > > > HMS > > > > > > > > > > pattern and would be happy to contribute it if I'm > > > successful. > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > Joy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
