WinkerDu opened a new pull request, #24830:
URL: https://github.com/apache/doris/pull/24830
## Proposed changes
Issue Number: close #xxx
I want to use Doris Multi-catalog to accelerate HMS query. My organization
has custom distributed file system, and we think wrapping the fs access
difference into broker (listLocatedFiles, openReader..) would be a elegant
approach.
This pr introduce HMS catalog conf `bind.broker.name`. If we set this conf,
file split, query scan operation will send to broker.
usage:
create a hms catalog with broker usage
```
CREATE CATALOG hive_catalog_broker PROPERTIES (
'type'='hms',
'hive.metastore.uris' = 'thrift://xxx',
'bind.broker.name' = 'hdfs_broker'
);
```
When we try to query from this catalog, file split and query scan request
will send to broker `hdfs_broker`.
More details about this pr:
1. Introduce HMS catalog proporty `bind.broker.name` to specify broker name
to do remote path work. When `bind.broker.name` is set, `enable.self.splitter`
must be `true` to ensure file splitting process is executed in Fe
2. Introduce 2 more interfaces to broker service:
- `TBrokerIsSplittableResponse isSplittable(1: TBrokerIsSplittableRequest
request)`, helps to invoke input format `isSplitable` interface.
- `TBrokerListResponse listLocatedFiles(1: TBrokerListPathRequest request)`,
helps to do `listFiles` or `listLocatedStatus` for remote file system
3. 3 parts of whole processing will be executed in broker:
- Check whether the path with specified input format name `isSplittable`
- `listLocatedFiles` of table / partition locations.
- `OpenReader` for specified file splits.
## Further comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
