Hello all, Thank you so much for all the feedback and guidance.
I am trying to summarize what I understood from the thread and some Slack conversations with JB and Prashant. *On how to structure the code:* JB suggested separate modules per catalog, like *extensions/federation/bqms*, *extensions/federation/glue*. Each module has its own factory class and manages its own dependencies, similar to how Hive federation works today if I understand correctly. Prashant suggested a hybrid, one generic factory that works with Iceberg's *BaseMetastoreCatalog* interface. Separate modules would only bring in dependencies, for example, iceberg-bigquery, no custom factory code would be needed per catalog. *On pluggability:* Dmitri prefers keeping things simple with current Quarkus module selection. Romain raised a good point about making it easier for end users who consume Polaris as a Docker image. But my understanding is limited here. *My plan:* I'd like to start with a simple approach since most people seem okay with that. I can try an initial implementation following JB's pattern i.e. separate module for BQMS or Prashant's pattern i.e. generic factory. I'd be okay if the first attempt doesn't get accepted, it may help move the discussion forward with some code to look at. Please let me know if I've misunderstood anything or if there's a preferred direction. Regards, Joy On Thu, 19 Mar, 2026, 4:52 am Romain Manni-Bucau, <[email protected]> wrote: > Hi, > > As an end users I have a small feedback on that topic: > > 1. it is unlikely we rebuild polaris to get federation (we'd federate > poalris to make it clear) - with hive catalog for now > 2. hive integration with polaris is not fully aligned with spark which has > shihms so there is already this classloader or alternative need > > > what can be envisionned if you do not want to go with classloaders is real > pluggability at *runtime* (like any real CDI application) or worse case > using a ServiceLoader (even if weird in a CDI stack since the SPI is CDI > impl itself). > > so from my small window a valid option is to make polaris runtime friendly > (vs build friendly which is not a solution for a product) and optionally > enable classloading (just needs to be a tree like > > https://github.com/Talend/component-runtime/blob/master/container/container-core/src/main/java/org/talend/sdk/component/container/ContainerManager.java > ). > the classloader support can even be made in a runtime extension which does > load a configuration to load other extension so you get the best of both > worlds, but point is you cant request users to *build* for a webapp > ultimately delivered as a docker image - I know apache is just about > sources but nobody consumes it this way (except vendors when they fork but > hopefully it is a few end users). > > hope it makes sense > > Romain Manni-Bucau > @rmannibucau <https://x.com/rmannibucau> | .NET Blog > <https://dotnetbirdie.github.io/> | Blog <https://rmannibucau.github.io/> > | Old > Blog <http://rmannibucau.wordpress.com> | Github > <https://github.com/rmannibucau> | LinkedIn > <https://www.linkedin.com/in/rmannibucau> | Book > < > https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064 > > > Javaccino founder (Java/.NET service - contact via linkedin) > > > Le mer. 18 mars 2026 à 23:18, Dmitri Bourlatchkov <[email protected]> a > écrit : > > > Hi All, > > > > Good point about modeling federation implementations similarly to > pluggable > > persistence implementations. > > > > I'm not sure about meddling with ClassLoaders in Polaris. I'd prefer to > > leverage the existing Quarkus-based integration of multiple modules into > a > > single runtime env. (current state). > > > > Downstream project will have the ability to elect which modules to use > > at build time. > > > > Cheers, > > Dmitri. > > > > On Wed, Mar 18, 2026 at 2:42 AM Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Hi everyone, > > > > > > "Classloader" is a fair and valid point. I have shared this perspective > > in > > > the past and discussed it with Nandor as well: I believe federation > > should > > > utilize a "plugin" mechanism, similar to how persistence is handled. > > > > > > By adopting this approach, each plugin (such as HMS) is in its own > > module, > > > and would be responsible for managing its own dependencies and shading. > > > > > > Regards, > > > JB > > > > > > On Wed, Mar 18, 2026 at 12:37 AM Yufei Gu <[email protected]> > wrote: > > > > > > > Federation seems like the strongest candidate given the dependency > > > surface. > > > > For example, Trino-style classloader isolation could help when > > conflicts > > > > arise (e.g., Hadoop or Guava across different federation > > > implementations). > > > > That said, I’d suggest introducing this only once we actually hit > such > > > > issues. Until then, keeping things simple with proper modularization > > may > > > be > > > > the better tradeoff. WDTY? > > > > > > > > Yufei > > > > > > > > > > > > On Tue, Mar 17, 2026 at 2:00 PM Madhan Neethiraj <[email protected]> > > > > wrote: > > > > > > > > > > we need to be careful about is proper modularization to control > > > > > dependency scope growth. > > > > > > > > > > I share this concern; it is applicable for any extension like > > > > authorizers. > > > > > Having a built-in support for isolation of extension specific > > > > dependencies > > > > > will significantly reduce/eliminate complexities in handling > > potential > > > > > library conflicts and avoid impact to Polaris core due to > > dependencies > > > > > dragged by extensions. Trino's plugin model seems to be a good > > > reference > > > > - > > > > > 1) and 2). > > > > > > > > > > Thanks, > > > > > Madhan > > > > > > > > > > 1) https://trino.io/docs/current/installation/plugins.html > > > > > 3) https://trino.io/docs/current/develop/spi-overview.html > > > > > > > > > > > > > > > On 3/17/26, 1:30 PM, "Prashant Singh via dev" < > > [email protected] > > > > > <mailto:[email protected]>> wrote: > > > > > > > > > > > > > > > +1 to this. I would also recommend thinking about this from an > > > interface > > > > > POV to have BaseMetastoreCatalog (from Apache Iceberg) as a way to > > > > > integrate, with a defined connection object that can feed the > > > connection > > > > to > > > > > BQ or Glue (perhaps a standard key-value prep) > > > > > and the dependencies can be added accordingly. > > > > > > > > > > > > > > > Thanks, > > > > > Prashant Singh > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2026 at 12:08 PM Dmitri Bourlatchkov < > > [email protected] > > > > > <mailto:[email protected]>> > > > > > wrote: > > > > > > > > > > > > > > > > Hi Joy, > > > > > > > > > > > > I suppose Polaris will be the REST API + AuthZ layer on top of > > > > > > BigQueryMetastoreCatalog, right? How do you envision this? > > > > > > > > > > > > In general, I think it's going to be a useful contribution. > > > > > > > > > > > > From my POV, the only technical issue we need to be careful about > > is > > > > > > proper modularization to control dependency scope growth. I > imagine > > > > some > > > > > > downstream projects may want to opt in/out of > > > BigQueryMetastoreCatalog > > > > > and > > > > > > its transitive dependencies at their own discretion. > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > On 2026/03/17 18:19:34 Joy wrote: > > > > > > > Hi, > > > > > > > > > > > > > > I'm Joy. We use BigQueryMetastoreCatalog for Iceberg on GCP and > > > want > > > > to > > > > > > use > > > > > > > Polaris as a catalog gateway. I see federation currently > supports > > > > REST > > > > > > and > > > > > > > HMS. > > > > > > > > > > > > > > Would there be interest in supporting other native Iceberg > > catalogs > > > > > like > > > > > > > BQMS or Glue? I've been working on a prototype for BQMS > following > > > the > > > > > HMS > > > > > > > pattern and would be happy to contribute it if I'm successful. > > > > > > > > > > > > > > Regards, > > > > > > > Joy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
