morningman opened a new issue, #67577:
URL: https://github.com/apache/doris/issues/67577
This issue tracks the work that makes MySQL and Arrow Flight SQL **equal
front ends over one session
layer** in the FE: one session object, one result-encoding abstraction, one
Doris-to-Arrow type
mapping, one connection pool and quota, and one prepared-statement registry.
Part of #65615. The design is being written up as a DSIP; this issue is the
implementation tracker.
If you hit one of the problems listed here, please open a separate bug
report and link it back.
**Relationship to the earlier protocol SPI work.** #60355 / #60361 (both
closed) decoupled the
*listeners*: how a protocol server is discovered, configured and started.
This issue covers
everything behind the listener - session state, the result path, type
mapping, connection governance
and prepared statements - which is still written against the MySQL protocol
and special-cased for
Arrow Flight SQL.
## Why
An Arrow Flight SQL session already *is* a `ConnectContext`, `SHOW
PROCESSLIST` lists it and `KILL`
works on it. What is not shared is most of what the execution layer does
afterwards. A read of the
FE session/connection/result paths and the BE result sinks turned up the
following; items with a PR
have been verified locally, the rest are code-reading conclusions and are
marked as such.
- The execution layer branches on the protocol in roughly 40 places
(`StmtExecutor` 18,
`ConnectProcessor` 8, nine `Command`s, plus `Coordinator`,
`NereidsCoordinator`,
`StatementContext`, `FrontendServiceImpl`) and reaches for `MysqlChannel`
in roughly 50. A Flight
session throws from `getMysqlChannel()`, so every unguarded reach is a
live failure rather than a
style problem - see #67569 for one that reached production.
- There are **three** Doris-to-Arrow type mappings: the BE's
`convert_to_arrow_type`, the FE's
`FlightSqlSchemaHelper.getArrowType` (used by `CommandGetTables`), and
`FlightSqlChannel`, which
encodes every FE-side result as `utf8`. The FE mirror already disagrees
with the BE in four types.
- Flight connections are governed by their own pool with their own limit
(`arrow_flight_max_connections`) plus a per-user token LRU, so
`qe_max_connection` and
`max_user_connections` do not apply to them, and a live session that the
LRU evicts surfaces to the
client as `invalid bearer token`.
- Flight prepared statements do not use the FE's prepared-statement
machinery:
`createPreparedStatement` stores the SQL text and returns a placeholder
schema, and parameter
binding is not implemented.
## Status convention
Same as #65615: `[x]` means merged or confirmed complete; `[ ]` means open
or needs follow-up.
## Stage 0 - independent bug fixes
Defects found by the survey that do not depend on the refactor. Submitted
2026-09-07; all three want
a branch pick label.
- [ ] **A Flight session cannot forward a statement to the master FE:**
#67569.
`FEOpExecutor.buildStmtForwardParams()` reads `CLIENT_DEPRECATE_EOF`
straight off the MySQL channel
(introduced by #61050), so on a multi-FE deployment every statement a
Flight connection has to
forward - any DDL issued to a follower or observer, or anything at all
under
`force_forward_all_queries` - fails with `getMysqlChannel not in mysql
connection`.
- [ ] **`ConnectType` declared twice, plus an unreachable dispatch on the
forward path:** #67572.
Cleanup, no behavior change.
- [ ] **A Flight session reports `0.0.0.0:0` as its client address:**
#67576. The `Host` column of
`SHOW PROCESSLIST` and `information_schema.processlist`, the audit log's
`client_ip`, and the kill
and timeout warnings all showed the placeholder, although the real address
is resolved when the
bearer token is issued.
## Stage 1 - one session, one result path
Behavior-preserving refactor, guarded by golden tests that pin the MySQL
packet bytes.
- [ ] Golden tests first: a recording `MysqlChannel` drives a statement set
through
`MysqlConnectProcessor`/`StmtExecutor` and the produced packets are
compared byte for byte; the same
statements then run through the Flight processor and their schema and rows
are compared. This is the
safety net for everything below, so it lands before any refactor.
- [ ] `ProtocolAdapter`: the per-protocol half of `ConnectContext` (channel,
capabilities, accept
loop, close, idle handling) moves behind an interface with a MySQL, a
Flight and an internal
implementation. `ConnectContext` keeps its class name and getter
signatures - it is referenced by
~900 files - and `FlightSqlConnectContext` goes away. Flight commands get
a per-session lock:
`ConnectContext` is not thread safe and Flight does not serialize a
session's calls today.
- [ ] `ResultSender`: the result-encoding half moves out of
`StmtExecutor`/`ConnectProcessor`, so the
three text-result paths (`SHOW`, `EXPLAIN`, replayed proxy results) all go
through one call.
- [ ] Remaining protocol branches in the execution layer go to zero: the
`Command`s, `Coordinator`,
`NereidsCoordinator`, `StatementContext`, the short-circuit rule and
`FEOpExecutor` switch to
capability predicates.
## Stage 2 - one type mapping
- [ ] One enumerable Doris-to-Arrow mapping, shared by a table-driven BE
unit test and an FE test that
validates the FE mirror against the same golden file, and the four known
FE/BE disagreements fixed.
- [ ] Field metadata completed on the BE: `doris_type` for every non-native
type plus
`ARROW:extension:name` (`doris.largeint`, `doris.ipv4`, `doris.ipv6`,
`doris.bitmap`, `doris.hll`,
`doris.quantile_state`, `doris.agg_state`, `doris.time`), with JSONB and
VARIANT using the canonical
`arrow.json` extension. **Storage types do not change**:
`convert_to_arrow_type` also serves the
Spark and Flink connectors' read path and Python UDFs, so changing them
would change those
protocols. In particular LARGEINT stays a decimal string rather than
becoming `decimal128(38,0)`,
which cannot represent an int128 (39 decimal digits; Arrow caps decimal128
precision at 38), and
IPV4/IPV6 stay `int32`/`utf8` rather than `fixed_size_binary`.
- [ ] FE-side results (`SHOW`, `EXPLAIN`, replayed proxy results) built as
typed vectors instead of
all-`utf8`.
- [ ] A cross-protocol regression suite: the same query over JDBC and over
Flight, compared column by
column across the whole type matrix.
## Stage 3 - one connection pool and quota
Builds on #67504 (merged 2026-09-07), which stopped a finished Flight query
from holding its
coordinator until `wait_timeout`.
- [ ] Single pool: `FlightSqlConnectPoolMgr` folds into `ConnectPoolMgr`,
and Flight registration goes
through `qe_max_connection` and `max_user_connections` like any other
connection.
- [ ] The bearer token becomes a session credential: no token cache, no
per-user LRU, no separate
lifetime. `arrow_flight_token_cache_size` and
`arrow_flight_token_alive_time_second` are deprecated
with a warning for one major version, and `arrow_flight_max_connections`
becomes a sub-quota that
defaults to following `qe_max_connection`.
- [ ] A `Protocol` column on `SHOW PROCESSLIST` and
`information_schema.processlist` (FE and the BE
schema scanner), plus a `protocol` field in the audit log.
## Stage 4 - prepared statements
- [ ] The prepared-statement registry is extracted from the MySQL
`COM_STMT_*` handling so both front
ends share it.
- [ ] Flight
`createPreparedStatement`/`getFlightInfoPreparedStatement`/`closePreparedStatement`
go
through it and return the real result and parameter schemas.
- [ ] `acceptPutPreparedStatementQuery` binds Arrow parameter batches, so
Flight SQL JDBC with
`useServerPrepStmts=true` and the ADBC default path work.
## Known gaps, not yet scheduled
Code-reading conclusions from the same survey. They are not blocking the
stages above; each needs its
own report or PR.
- [ ] **A forwarded statement's result set is dropped over Flight.** Even
after #67569, a statement
that is forwarded to the master and returns rows gives the Flight client
nothing, because the proxy
result is replayed by `ConnectProcessor.finalizeCommand()`, which is
MySQL-only. Affects only
statements that both need forwarding and carry a result set.
- [ ] **`arrow_flight_token_alive_time_second` is applied in the wrong
unit.** `createToken()` converts
the configured seconds as if they were minutes, so the recorded expiry is
60x too far out; the token
is actually evicted by the Guava cache's own expiry, which makes the
explicit expiry check dead code.
- [ ] **`AGG_STATE` is advertised as `utf8`** while carrying non-UTF-8 bytes
- the same shape as the
Iceberg `BINARY` problem in #67371.
- [ ] **`select 1` and `select @@var` always go to the BE on a Flight
session**, because
`supportHandleByFe()` is hard-coded false for Arrow Flight SQL. Enabling
FE-side results for Flight
is deliberately out of scope until the FE result path is typed (Stage 2).
## Upstream PRs this work depends on
- [x] #67504 - release a finished Flight query's coordinator instead of
holding it until
`wait_timeout`. Merged 2026-09-07; unblocks Stage 1.
- [ ] #67530 - keep the Doris type of a nested LARGEINT in the Arrow schema.
Under a requested BE
type-layer refactor; Stage 2 waits on the outcome so the two do not
collide.
- [ ] #65789 - utf8/large-string schema-array mismatch for oversized
columns. Stage 2 rebases on it
rather than touching `arrow_block_convertor.cpp`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01CCAMVmoWN2MGVVsSRDCf2p
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
