> the resolution scope of a datasource must stay consistent with the maximum transaction scope (realm sounds okish), something finer grained can lead to inconsistencies until you do setup XA which is not that trivial with quarkus OOTB
Exactly. I'm less worried about events and metrics related to the transaction scope though. Besides the transaction scope, we must be careful about whether a join is needed across tables, which is sometimes impossible if those tables are in different data sources. For example, the current metrics tables only have the table entity id, which has to be joined with the entity table to make sense of it. Yufei On Thu, Mar 12, 2026 at 12:39 AM Romain Manni-Bucau <[email protected]> wrote: > Hi, > > two small notes on the PR from an end user standpoint: > > 1. to make it usable by the community (vs vendors who can always mutate > beans at build time) it should have a default implementation which is > configurable - vs doesn't add a feature compared to today - with MP config > (in the spirit same capability than > > https://tomee.apache.org/tomee-10.1/examples/dynamic-datasource-routing.html > but using another config mechanism) > 2. the resolution scope of a datasource must stay consistent with the > maximum transaction scope (realm sounds okish), something finer grained can > lead to inconsistencies until you do setup XA which is not that trivial > with quarkus OOTB > > hope it helps > > Romain Manni-Bucau > @rmannibucau <https://x.com/rmannibucau> | .NET Blog > <https://dotnetbirdie.github.io/> | Blog <https://rmannibucau.github.io/> > | Old > Blog <http://rmannibucau.wordpress.com> | Github > <https://github.com/rmannibucau> | LinkedIn > <https://www.linkedin.com/in/rmannibucau> | Book > < > https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064 > > > Javaccino founder (Java/.NET service - contact via linkedin) > > > Le jeu. 12 mars 2026 à 07:29, EJ Wang <[email protected]> a > écrit : > > > Hi Subham, Dmitri, Yufei, JB, > > > > I’m generally aligned with the direction here, and I left a few more > > detailed comments on the PR. > > At a high level, my main concern is that the current proposal may still > be > > a bit ahead of the current story/scope. I’d lean toward keeping the first > > step narrower, preferring purpose-based routing over per-realm routing > for > > v1, and making the supported model/config+migration story more explicit > > before the broader contract hardens. > > > > -ej > > > > On Wed, Mar 11, 2026 at 12:37 PM Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Hi Subham, > > > > > > Thanks for this contribution. It's an interesting feature. > > > > > > As mentioned in the GitHub issues, I am fine with moving forward with a > > PR > > > as long as it remains a Draft PR to help drive the discussion. I > suggest > > > linking a GitHub Discussion or a Design Doc within the PR to help build > > > consensus. > > > > > > That said, I have a few initial comments: > > > > > > 1. I like the SPI approach used in the PR. This should become a > standard > > in > > > Polaris to facilitate custom implementations. > > > 2. I agree that having a data source "per purpose" is a good idea. The > > main > > > question is how we should handle the split: > > > - Very granular (per entity) > > > - By table "meaning" > > > - By realm (this may not be granular enough) > > > > > > From a user standpoint, I believe we should keep it simple. It would > be a > > > first great step forward. For example, in the configuration (e.g., > > > application.properties), it could look like: > > > - polaris.datasource.entities= > > > - polaris.datasource.events= > > > - polaris.datasource.grants= > > > > > > Regards, > > > JB > > > > > > On Tue, Mar 10, 2026 at 11:19 PM Yufei Gu <[email protected]> > wrote: > > > > > > > Hi Subham, > > > > > > > > Thanks for working on this. Given the complexity and long term > > > implications > > > > discussed in https://github.com/apache/polaris/issues/3890, I think > a > > > > short > > > > design doc could still be helpful to capture the intended > architecture > > > and > > > > future evolution. Here are a few questions listed in the issue. I > > believe > > > > these should be answered before jumping to an implementation. > > > > > > > > > > > > 1. Should we split each potential noisy table into its own > dedicated > > > > data source. For example, one data source for events, one for > > metrics, > > > > and > > > > one for idempotency. > > > > 2. Should we allow flexible grouping. For example, events and > > > > idempotency tables sharing one data source, while metrics uses > > > another. > > > > 3. Should we consider different DS per realm instead of > table-level > > > > spliting? > > > > 4. How should schema version information be managed. If tables > live > > in > > > > different data sources, how do we track and coordinate schema > > > evolution. > > > > 5. Should different data sources be allowed to point to different > > > > schemas or databases. This likely aligns with the isolation goal, > > but > > > it > > > > implies that cross table joins become difficult or impossible at > the > > > > database level, leaving only in memory joins as an option. > > > > 6. Should different data sources be allowed to point to the same > > > schema. > > > > If not, we need validation logic to detect and prevent > > > misconfiguration. > > > > > > > > > > > > Yufei > > > > > > > > > > > > On Tue, Mar 10, 2026 at 7:33 AM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > Hi Subham, > > > > > > > > > > Thanks again for your contribution! > > > > > > > > > > I believe PR 3960 moves in the right direction by establishing an > SPI > > > to > > > > > delegate DataSource resolution logic to the runtime environment. > > > > > > > > > > It immediately allows custom implementations in downstream projects > > (if > > > > > people wish to do that) and opens a way for supporting multiple > > > > DataSources > > > > > in Apache Polaris (in follow-up PRs), > > > > > > > > > > I think the PR is pretty clear in itself and does not require any > > extra > > > > > design docs. Let's review it in GH and merge when we have > consensus. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Tue, Mar 10, 2026 at 8:27 AM Subham Sangwan < > > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi Polaris Dev Team I have opened PR #3960 [1] to introduce the > > > > > > foundational groundwork for multi-datasource support in JDBC > > > > persistence, > > > > > > addressing Issue #3890 [2].The goal is to enable physical > isolation > > > of > > > > > > different persistence workloads (METASTORE, METRICS, EVENTS) into > > > > > dedicated > > > > > > connection pools or databases. This will allow Polaris to better > > > handle > > > > > > high-traffic environments by preventing "noisy neighbor" effects > on > > > the > > > > > > core entity tables. > > > > > > > > > > > > Key Highlights: > > > > > > > > > > > > - DataSourceResolver: A new pluggable interface for routing > JDBC > > > > > > connections based on RealmContext and StoreType. > > > > > > - Modular Design: Decoupled the resolution implementation into > > the > > > > > > runtime-common module. > > > > > > - Consistency: Utilizes a type-safe StoreType enum and aligns > > with > > > > > > existing RealmContext patterns. > > > > > > > > > > > > The PR has been refined with feedback from @dimas-b and is now > > ready > > > > for > > > > > > community review. I'd appreciate any feedback on the overall > > > approach. > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Subham Sangwan > > > > > > GitHub: Subham-KRLX > > > > > > > > > > > > [1] https://github.com/apache/polaris/pull/3960 > > > > > > [2] https://github.com/apache/polaris/issues/3890 > > > > > > > > > > > > > > > > > > > > >
