morningman commented on issue #65479:
URL: https://github.com/apache/doris/issues/65479#issuecomment-4944729641
## Design note: prevent expensive FE-side metadata scans
One important risk in this migration is that some `information_schema`
tables are populated by FE metadata, but the SQL plan still runs through normal
BE execution.
For example, the old command:
```sql
SHOW PARTITIONS FROM table1;
```
has a natural table context. FE only needs to read partition metadata for
`table1`.
After replacing it with a system table query, users may write:
```sql
SELECT * FROM information_schema.partitions WHERE table_name = 'table1';
```
If the predicate is only evaluated in BE, the BE must first fetch partition
metadata for many tables from FE, then apply the filter locally. For
high-cardinality metadata such as partitions, tablets, rowsets, or table
properties, this can put avoidable pressure on FE.
### Current code shape
Doris already has part of the required infrastructure:
- `PushDownFilterIntoSchemaScan` extracts fixed predicates on
`TABLE_CATALOG`, `TABLE_SCHEMA`, and `TABLE_NAME` from `LogicalSchemaScan`
filters.
- `LogicalSchemaScan` and `PhysicalSchemaScan` already carry
`schemaCatalog`, `schemaDatabase`, `schemaTable`, and `frontendConjuncts`.
- `SchemaScanNode` already sends `db`, `table`, and `frontend_conjuncts` to
BE.
- However, for `information_schema.partitions`, `SchemaPartitionsScanner`
currently sends only `catalog` and `dbId` back to FE in
`TSchemaTableRequestParams`.
- `MetadataGenerator.partitionsMetadataResult` then calls
`database.getTables()` and scans all tables in the database.
So the existing `TABLE_NAME = 'table1'` filter can be seen by the planner,
but it is not yet used to reduce the data generated by FE for
`information_schema.partitions`.
### Recommended approach
The safest design is not only to require predicates. We should combine two
mechanisms:
1. Add a planner-side guard for high-cardinality system tables.
2. Push the extracted constraints all the way to FE so FE can generate less
metadata.
The guard protects FE from accidental full scans. The pushdown preserves the
value of SQL access by making filtered queries efficient.
### Planner-side guard
Add a validation step after schema-scan predicate extraction. This step
should reject full scans for selected high-cardinality system tables unless the
query contains a required, pushdown-capable constraint.
For example:
```text
information_schema.partitions:
require one of:
- TABLE_SCHEMA + TABLE_NAME
- TABLE_CATALOG + TABLE_SCHEMA + TABLE_NAME
information_schema.tablets / rowsets:
require one of:
- TABLE_SCHEMA + TABLE_NAME
- TABLE_ID
- TABLET_ID, if the table supports it
small metadata tables:
allow full scan
```
This should be table-specific. We should avoid hard-coding everything in one
rule. A small registry is easier to maintain, for example:
```text
partitions:
risk: HIGH
requiredAny:
- [TABLE_SCHEMA, TABLE_NAME]
pushdownKeys:
- TABLE_CATALOG
- TABLE_SCHEMA
- TABLE_NAME
- PARTITION_NAME
workload_groups:
risk: LOW
fullScanAllowed: true
```
When a new metadata table is added under `information_schema`, the developer
should define whether full scan is allowed and which predicates can be pushed
to FE.
### Predicate extraction
Start with simple, reliable predicates:
- `TABLE_CATALOG = literal`
- `TABLE_SCHEMA = literal`
- `TABLE_NAME = literal`
- `literal = TABLE_NAME`
- optionally `TABLE_NAME IN (...)`, with a limit on the number of values
More complex predicates such as `lower(table_name) = 't1'` or join-derived
predicates should not be required for the first version. They can still be
evaluated in BE after FE returns the reduced row set.
### FE-side pushdown
For `information_schema.partitions`, the minimal end-to-end change should be:
1. Let `PushDownFilterIntoSchemaScan` extract `TABLE_SCHEMA` and
`TABLE_NAME`.
2. Pass the extracted `table` value from `SchemaScanNode` to
`SchemaPartitionsScanner`.
3. Extend `TSchemaTableRequestParams` with a structured table filter, for
example `optional string table`, instead of relying on serialized
`frontend_conjuncts` for this core path.
4. In `MetadataGenerator.partitionsMetadataResult`, if the table filter is
set, fetch only that table instead of calling `database.getTables()`.
5. Keep the remaining SQL predicate in BE so behavior stays correct even if
only part of the predicate is pushed down.
This gives us a small and testable first step: `SELECT * FROM
information_schema.partitions WHERE table_schema = 'db1' AND table_name =
'table1'` should make FE inspect only `table1`.
### Why not only reject full scans?
Rejecting full scans is useful, but it is not enough by itself. If the query
is accepted but the predicate is not pushed into FE metadata generation, BE
still fetches too much data from FE. The real requirement is: high-cardinality
metadata tables must have both a required constraint and an FE-side
implementation that uses that constraint.
### Summary
For high-cardinality `information_schema` metadata tables, the migration
should use this contract:
- The planner extracts supported constraints from SQL predicates.
- A validation rule rejects unsafe full scans for selected tables.
- BE forwards structured constraints to FE.
- FE uses those constraints to reduce metadata enumeration.
- BE still evaluates the original SQL predicate for correctness.
`information_schema.partitions` is a good first target because it already
exposes the problem clearly and has a natural table-level filter equivalent to
`SHOW PARTITIONS FROM <table>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
