Doris-Breakwater commented on issue #67368: URL: https://github.com/apache/doris/issues/67368#issuecomment-5490609739
Breakwater-GitHub-Analysis-Slot: slot_f074cb282cd2 ## Initial assessment This is a confirmed server-side Arrow Flight SQL compatibility bug in Apache Doris 4.1.3-rc02 (`31263df4dc1d4d3a27517d264802cd4d6b92c874`), rather than an ADBC-only error. The report is sufficiently detailed for implementation: the exact commit and deterministic same-connection A/B test isolate the failure to the short-circuit execution mode, and the source at that commit explains the empty endpoint list. Impact is limited to queries that are eligible for the point-query short-circuit path and are executed through Arrow Flight SQL. It prevents results from being fetched, but there is no evidence here of data loss or incorrect data. The stated `enable_short_circuit_query=false` workaround is valid as a temporary, explicit workaround. I did not independently start a 4.1.3-rc02 cluster; the conclusion below is based on the supplied reproducer plus a trace of the exact reported source revision. ## Verified code path and root cause 1. `DorisFlightSqlProducer.executeQueryStatement()` clears prior endpoint locations, executes the statement, and for a remote result immediately calls `fetchArrowFlightSchema()` before constructing `FlightInfo`: [DorisFlightSqlProducer.java#L187-L284](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlProducer.java#L187-L284). 2. A Flight SQL SELECT sets `returnResultFromLocal=false`. When Nereids marks the query as short-circuit eligible, `StmtExecutor.executeAndSendResult()` selects `PointQueryExecutor`, calls `exec()`, then takes the Arrow Flight branch and returns before the normal `getNext()` loop: [StmtExecutor.java#L1381-L1439](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java#L1381-L1439). 3. `PointQueryExecutor.exec()` is intentionally a no-op; the tablet lookup and row retrieval happen only in `getNext()`: [PointQueryExecutor.java#L247-L300](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/qe/PointQueryExecutor.java#L247-L300). Therefore the Flight branch returns before the point lookup is even requested. 4. Endpoint locations are registered by the normal `Coordinator` and `NereidsCoordinator` result-sink paths ([Coordinator.java#L853-L886](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java#L853-L886), [NereidsCoordinator.java#L463-L484](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/qe/NereidsCoordinator.java#L463-L484)). `PointQueryExecutor` does not create a result sink or call `addFlightSqlEndpointsLocation()`. 5. Consequently, `FlightSqlConnectProcessor.fetchArrowFlightSchema()` reaches its deterministic empty-list guard and throws the exact reported error: [FlightSqlConnectProcessor.java#L112-L116](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/FlightSqlConnectProcessor.java#L112-L116). This is more than missing endpoint metadata. The point-query RPC response is explicitly a MySQL row batch (`is_binary_row` in [internal_service.proto#L378-L406](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/gensrc/proto/internal_service.proto#L378-L406)), and the BE path uses `VMysqlResultWriter`: [point_query_executor.cpp#L645-L666](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/be/src/service/point_query_executor.cpp#L645-L666). Adding a `FlightSqlEndpointsLocation` alone would advertise an endpoint for which no Flight result stream or Arrow schema exists. Both direct and prepared-statement FlightInfo paths call the same `executeQueryStatement()` method, so the gap is not specific to one ADBC statement mode. ## Missing information No information is blocking initial triage or root-cause confirmation. For the interoperability/version matrix only, it would be useful—but not required for the fix—to record the installed versions of `adbc_driver_flightsql`, `adbc_driver_manager`, and `pyarrow`. An FE stack trace/query ID would provide runtime confirmation of the guard above, but should not be required before accepting this issue. The issue currently has no labels, assignee, milestone, or linked development work. Suggested labels are `kind/bug` and `area/rpc`, consistent with the Arrow Flight SQL tracking issue. ## Recommended next steps 1. **Correctness-first fix:** prevent `LogicalResultSinkToShortCircuitPointQuery` from selecting the short-circuit mode for Arrow Flight SQL connections, so these queries use the existing coordinator/result-sink path and produce a valid Flight endpoint. The eligibility decision is made here: [LogicalResultSinkToShortCircuitPointQuery.java#L63-L89](https://github.com/apache/doris/blob/31263df4dc1d4d3a27517d264802cd4d6b92c874/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/LogicalResultSinkToShortCircuitPointQuery.java#L63-L89). This is the smallest safe server-side compatibility fix, at the cost of the point-query optimization for Flight clients. 2. **Full support alternative:** define an Arrow-native result transport for point lookup (schema plus data and lifecycle), either through a real BE Flight stream/endpoint or an FE-local `VectorSchemaRoot`. This is a larger change because the existing response is MySQL-encoded. Do not implement an endpoint-only patch. 3. Add regression coverage under `arrow_flight_sql_p0`, using this UNIQUE KEY MOW row-store table. Cover both a matching key and an empty result with short-circuit enabled by default, plus the disabled-short-circuit control. If the correctness-first gate is chosen, add an FE rule test proving Flight connections are not marked short-circuit; if full support is chosen, add a test proving the optimized path itself was used. 4. Evaluate a branch-4.1 backport after the fix is verified, since the report is against 4.1.3-rc02. Keep the explicit workaround documented until a fixed build is available. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
