wenzhenghu opened a new issue, #66025:
URL: https://github.com/apache/doris/issues/66025
## Summary
In the external session-bypass path, `ExternalDatabase.getTables()` performs
**N+1 remote table-name enumerations**.
When `extCatalog.shouldBypassTableNameCache(sessionContext)` is true:
1. `getTables()` calls `getTableNamesWithLock()`, which enumerates remote
table names once via `listLocalTableNamesWithoutCache(sessionContext)`.
2. Then `getTables()` iterates all table names and calls
`getTableNullable(tblName)`.
3. In bypass mode, `getTableNullable(tblName)` enters
`getTableNullableWithoutCache(sessionContext, tableName)`.
4. `getTableNullableWithoutCache(...)` calls
`findTableNamePairWithoutCache(...)`, which again does a full
`listTableNames(sessionContext)` scan for each table.
As a result, listing `N` tables triggers `1 + N` remote `LIST TABLES`
enumerations.
## Affected code
`fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java`
Current structure:
```java
public List<T> getTables() {
Set<String> tblNames = getTableNamesWithLock(); // 1 full remote
enumeration in bypass mode
for (String tblName : tblNames) {
T tbl = getTableNullable(tblName); // each iteration
re-enters bypass path
}
}
public Set<String> getTableNamesWithLock() {
if (extCatalog.shouldBypassTableNameCache(sessionContext)) {
return
Sets.newHashSet(listLocalTableNamesWithoutCache(sessionContext));
}
}
private T getTableNullableWithoutCache(SessionContext sessionContext, String
tableName) {
Optional<Pair<String, String>> matched =
findTableNamePairWithoutCache(sessionContext, tableName);
// findTableNamePairWithoutCache() -> listTableNames(sessionContext)
}
```
## Impact
This affects session-bypass catalogs such as delegated Iceberg REST session
catalogs.
For `SHOW TABLES` / `getTables()` on a database with many tables, the
current implementation scales poorly because every table object construction
repeats a full remote name enumeration.
The issue is performance-only, but the overhead can become significant for
large databases or expensive remote catalogs.
## Scope / origin
This is a **pre-existing master/baseline issue**, not introduced by PR
#65126.
The bypass structure already existed before the external meta cache
refactor. PR #65126 carried it forward and added related session-bypass
improvements elsewhere, but did not address this `getTables()` path.
## Suggested fix
In bypass mode, reuse a **single remote name enumeration** inside
`getTables()`:
- enumerate `(remoteName, localName)` pairs once,
- build a local lookup map from the result,
- construct table objects directly from that snapshot instead of re-calling
`getTableNullable()` for every table.
This keeps bypass semantics unchanged while removing the repeated full scans.
## Related
- PR #65126
- Commit `48291325300 [fix](fe) Avoid unnecessary session table enumeration`
addressed a nearby delegated-session existence-check path, but not this
`getTables()` N+1 path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
