This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new a00f8c59e6e [feature](lance) Support OSS storage provider (#67157)
a00f8c59e6e is described below
commit a00f8c59e6ed91a71ae957ce35092a7e2e1f93bf
Author: FANNG <[email protected]>
AuthorDate: Mon Sep 7 10:55:13 2026 +0800
[feature](lance) Support OSS storage provider (#67157)
### What problem does this PR solve?
Issue Number: close #67131
Related PR: #66805
Problem Summary:
Lance catalogs already forward provider-native storage options, but
Doris did not translate its typed OSS properties, and filesystem
catalogs rejected `oss://` warehouses.
This change:
- maps Doris OSS properties to Lance's public OSS options, including
anonymous access and addressing style;
- normalizes Namespace-vended aliases so the two spellings of one option
cannot reach Lance as competing entries, rejecting conflicting values
and passing unknown options through untouched;
- reads a vended option the way the store will: aliases are matched
case-insensitively, since OpenDAL lower-cases every config key before
deserializing, and the anonymous flag is parsed with OpenDAL's own
boolean grammar (`true|on` / `false|off`) rather than Java's, so a
vended `ACCESS_KEY_ID` or `allow_anonymous=on` cannot end up with a
credential configured and every request sent unsigned;
- separates translation from inference, so options derived from another
option - `allow_http` from the effective S3 endpoint, the OSS anonymous
flag from the effective credential - are computed once from the final
merged configuration rather than from the catalog's half alone;
- accepts `oss://` filesystem warehouses, requiring one that names a
bucket, and refuses OSS-HDFS - recognized with the same predicate Doris
routes on - since Doris reads that form through its HDFS-compatible
properties, which carry no Lance OSS storage options; and
- redacts OSS credentials from every provider-facing failure.
It reuses the FE-to-BE storage-option transport introduced by #66805 and
changes no Thrift or BE code.
### Release note
Support native Alibaba Cloud OSS storage options for Lance catalogs.
Two derived options now follow the final merged configuration instead of
the catalog's half alone. For S3, `allow_http` is decided by the
endpoint actually in effect, so a namespace that vends a plain-HTTP
endpoint no longer fails to reach it, and one that vends an HTTPS
endpoint no longer carries a stale permission. For OSS, a dataset the
catalog holds no storage configuration for is now explicitly marked
anonymous rather than leaving that key absent; a deployment that relied
on an exported `OSS_ALLOW_ANONYMOUS` to decide it for such a catalog
should configure the catalog instead.
### Check List (For Author)
- Test:
- Unit Test: Official Doris build image `run-fe-ut.sh`; locally 94 tests
across the Lance and storage-property suites.
- Manual test: End-to-end scans against a real Alibaba Cloud OSS bucket
on a native arm64 cluster. Every case returns `count(*) = 4, sum(amount)
= 100` with matching full rows, type mapping (`bigint`/`text`/`int`) and
predicate pushdown:
- **dir namespace** (`lance.catalog.type=filesystem` →
`LanceNamespace.connect("dir")`) over an `oss://` warehouse with Doris
OSS properties;
- **REST namespace** against a real **Apache Gravitino 1.3.0**
`lance-rest` service with **no static OSS credentials on the Doris
catalog at all** — Gravitino vends the bare OSS-native spellings
`endpoint` / `access_key_id` / `access_key_secret` / `region`, which
this PR normalizes before handing them to the BE, so the scan succeeding
is what shows the vended path works;
- a warehouse with no bucket (`oss:/path`) is rejected at `CREATE
CATALOG`, and the qualified `oss://bucket.<endpoint>/path` form is
passed to the engine as written rather than rewritten - the store then
reports the host it actually tried, `bucket.<endpoint>.<endpoint>`;
- **STS temporary credentials**: an `AssumeRole` triple scans, and the
same temporary key pair *without* `oss.session_token` fails with OSS's
own `InvalidAccessKeyId` / "The Security Token may be lost to specify
that it is a STS Access Id" — the negative control that shows the token
reaches the BE rather than the scan succeeding by another route;
- **anonymous flag vs. vended credentials**: a catalog with
`oss.endpoint`/`oss.region` but no key pair reads as anonymous on its
own, and still scans once Gravitino vends a credential. This is the case
the plain Gravitino test cannot reach, because a catalog with no `oss.*`
at all never builds an OSSProperties and so never claims anonymity.
Two more negative controls: a Gravitino catalog vending both `endpoint`
and `oss_endpoint` with different values is rejected; a vended half key
pair is rejected rather than silently paired with the catalog's other
half.
Query profiles confirm the scans ran on the BE, not the FE: splits
assigned to the backend and `FILE_SCAN_OPERATOR(table_name=all_types)`
reporting `RowsProduced: sum 4`. The BE is a stock arm64 build of the
same `branch-4.1` base — this PR changes no BE code, and `gensrc/thrift`
and `gensrc/proto` are byte-identical between the two trees.
- Build: Full FE build and Checkstyle passed.
- Behavior changed: Yes. Lance catalogs can use `oss://` with Doris OSS
properties and with OSS options vended by a Namespace.
- Does this need documentation: Yes — apache/doris-website#4094 adds an
OSS section to the lance-catalog page, in both the English and zh-CN
copies.
### Scope
Not included: role-based credential vending, STS credential refresh
during a scan, a new Thrift envelope, or static support for other
object-store providers.
Namespace-vended options overlay the catalog's key by key, so a
namespace is expected to vend a complete credential rather than part of
one. An earlier revision of this PR replaced the credential as a unit;
it was removed as more machinery than the cases it guarded against
warranted. Where a vended anonymous flag meets a configured credential
the two are reported as a conflict rather than one side quietly winning.
`LanceS3StorageProvider` is left alone. Its comparable defects predate
this work (#66805) and are better addressed for all providers at once
than widened into an OSS change.
Anonymous OSS access itself is covered by unit tests only — how the flag
is inferred is exercised end to end, but issuing a genuinely unsigned
read needs an anonymously readable bucket, which I do not have. Its
correctness rests on lance forwarding unrecognized options to OpenDAL
plus OpenDAL's OSS config accepting the flag (`OssCore::sign` returns
the request unsigned whenever it is set).
---
.../datasource/lance/LanceExternalCatalog.java | 12 +-
.../datasource/lance/LanceMetadataLoader.java | 2 +-
.../datasource/lance/LanceOssStorageProvider.java | 185 ++++++++
.../lance/LancePassThroughStorageProvider.java | 22 +-
.../datasource/lance/LanceS3StorageProvider.java | 26 +-
.../datasource/lance/LanceStorageOptions.java | 42 +-
.../datasource/lance/LanceStorageProvider.java | 31 +-
.../LanceFileSystemMetastoreProperties.java | 35 +-
.../doris/datasource/tvf/source/TVFScanNode.java | 2 +-
.../ExternalFileTableValuedFunction.java | 2 +-
.../lance/LanceFilesystemCatalogTest.java | 12 +-
.../datasource/lance/LanceStorageOptionsTest.java | 504 +++++++++++++++++++--
.../property/metastore/LancePropertiesTest.java | 46 ++
13 files changed, 842 insertions(+), 79 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java
index cef2b064b98..4a4a8111287 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java
@@ -78,7 +78,11 @@ public class LanceExternalCatalog extends ExternalCatalog {
private static final long ALLOCATOR_LIMIT = 256L * 1024 * 1024;
private static final int MAX_PROVIDER_MESSAGE_BYTES = 1024;
private static final String[] RUNTIME_SENSITIVE_OPTION_KEYS = {
- "aws_access_key_id", "aws_secret_access_key", "aws_session_token"
+ "aws_access_key_id", "aws_secret_access_key", "aws_session_token",
+ // OSS credentials only reach these options because of this
change, so they have to be
+ // recognized here too. The emitted spelling is the only one that
occurs: the map read
+ // below is the merged one, where a vended alias has already been
normalized onto it.
+ "oss_access_key_id", "oss_secret_access_key", "oss_security_token"
};
private transient LanceNamespace namespace;
@@ -104,7 +108,7 @@ public class LanceExternalCatalog extends ExternalCatalog {
rootDatabase = properties.getRootDatabase();
parentNamespace = LanceNamespaceName.parseParentNamespace(
properties.getNamespaceParent(),
properties.getNamespaceDelimiter());
- namespaceStorageOptions = LanceStorageOptions.forUri(
+ namespaceStorageOptions =
LanceStorageOptions.fromDorisStorageProperties(
properties.getNamespaceStorageUri(),
catalogProperty.getOrderedStoragePropertiesList());
@@ -127,7 +131,7 @@ public class LanceExternalCatalog extends ExternalCatalog {
}
AbstractLanceProperties properties = getLanceProperties();
- Map<String, String> storageOptions = LanceStorageOptions.forUri(
+ Map<String, String> storageOptions =
LanceStorageOptions.fromDorisStorageProperties(
properties.getNamespaceStorageUri(),
catalogProperty.getOrderedStoragePropertiesList());
List<String> parent = LanceNamespaceName.parseParentNamespace(
@@ -434,7 +438,7 @@ public class LanceExternalCatalog extends ExternalCatalog {
// One option map serves both readers: the FE opens the dataset
through the Lance Java SDK
// and the BE through lance-c, so neither can end up with credentials
the other lacks. The
// dataset URL picks the option vocabulary, the same way Lance picks a
provider from it.
- Map<String, String> storageOptions =
LanceStorageOptions.forVendedTable(datasetUri,
+ Map<String, String> storageOptions =
LanceStorageOptions.fromDorisAndVendedStorageOptions(datasetUri,
catalogProperty.getOrderedStoragePropertiesList(),
table.getStorageOptions());
return new ResolvedTableAccess(datasetUri, storageOptions);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceMetadataLoader.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceMetadataLoader.java
index 9c4803af710..38c32c3de35 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceMetadataLoader.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceMetadataLoader.java
@@ -54,7 +54,7 @@ public final class LanceMetadataLoader {
throws Exception {
try (BufferAllocator allocator = new RootAllocator(ALLOCATOR_LIMIT)) {
return loadLatest(datasetUri,
- LanceStorageOptions.forUri(datasetUri, storageProperties),
allocator);
+ LanceStorageOptions.fromDorisStorageProperties(datasetUri,
storageProperties), allocator);
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceOssStorageProvider.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceOssStorageProvider.java
new file mode 100644
index 00000000000..d51f9765bfd
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceOssStorageProvider.java
@@ -0,0 +1,185 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.lance;
+
+import org.apache.doris.datasource.property.storage.OSSProperties;
+import org.apache.doris.datasource.property.storage.StorageProperties;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+/** Alibaba Cloud OSS storage, which Lance reaches through its OpenDAL OSS
provider. */
+final class LanceOssStorageProvider implements LanceStorageProvider {
+
+ static final LanceOssStorageProvider INSTANCE = new
LanceOssStorageProvider();
+
+ private static final String ENDPOINT = "oss_endpoint";
+ private static final String ACCESS_KEY_ID = "oss_access_key_id";
+ private static final String SECRET_ACCESS_KEY = "oss_secret_access_key";
+ private static final String REGION = "oss_region";
+ private static final String SECURITY_TOKEN = "oss_security_token";
+ private static final String ADDRESSING_STYLE = "addressing_style";
+ private static final String SKIP_SIGNATURE = "skip_signature";
+ private static final String ALLOW_ANONYMOUS = "allow_anonymous";
+
+ /**
+ * Lance exposes the {@code oss_*} names as its public storage-option
vocabulary and normalizes
+ * them to OpenDAL's field names before constructing the operator. Both
spellings are accepted,
+ * so collapse only these known pairs before merging static and
namespace-vended options.
+ */
+ private static final Map<String, String> PUBLIC_BY_ALIAS =
ImmutableMap.<String, String>builder()
+ .put("endpoint", ENDPOINT)
+ .put(ENDPOINT, ENDPOINT)
+ .put("access_key_id", ACCESS_KEY_ID)
+ .put(ACCESS_KEY_ID, ACCESS_KEY_ID)
+ .put("access_key_secret", SECRET_ACCESS_KEY)
+ .put(SECRET_ACCESS_KEY, SECRET_ACCESS_KEY)
+ .put("region", REGION)
+ .put(REGION, REGION)
+ .put("security_token", SECURITY_TOKEN)
+ .put(SECURITY_TOKEN, SECURITY_TOKEN)
+ // OpenDAL 0.57 renamed this option to skip_signature, but lance-c
0.1.6 still uses
+ // OpenDAL 0.56. Emit the old spelling, which newer OpenDAL keeps
as an alias.
+ .put(SKIP_SIGNATURE, ALLOW_ANONYMOUS)
+ .put(ALLOW_ANONYMOUS, ALLOW_ANONYMOUS)
+ .build();
+
+ private LanceOssStorageProvider() {
+ }
+
+ @Override
+ public Map<String, String> normalizeDorisStorageOptions(
+ List<StorageProperties> storageProperties) {
+ Map<String, String> result = new HashMap<>();
+ OSSProperties properties = selectOss(storageProperties);
+ if (properties == null) {
+ return result;
+ }
+ putIfNotEmpty(result, ENDPOINT, properties.getEndpoint());
+ putIfNotEmpty(result, REGION, properties.getRegion());
+ putIfNotEmpty(result, ACCESS_KEY_ID, properties.getAccessKey());
+ putIfNotEmpty(result, SECRET_ACCESS_KEY, properties.getSecretKey());
+ putIfNotEmpty(result, SECURITY_TOKEN, properties.getSessionToken());
+
+ // Lance snapshots the host's OSS_*/AWS_*/ALIBABA_CLOUD_* environment
into the same config
+ // map before storage options are applied, so state both addressing
styles explicitly the
+ // way the S3 provider does. Leaving the default implicit would let an
exported
+ // OSS_ADDRESSING_STYLE outrank an explicit oss.use_path_style=false.
+ String usePathStyle = properties.getUsePathStyle();
+ if (StringUtils.isNotEmpty(usePathStyle)) {
+ result.put(ADDRESSING_STYLE, Boolean.parseBoolean(usePathStyle) ?
"path" : "virtual");
+ }
+ return result;
+ }
+
+ @Override
+ public Map<String, String> normalizeVendedStorageOptions(
+ Map<String, String> vendedOptions) {
+ Map<String, String> result = new HashMap<>();
+ if (vendedOptions == null) {
+ return result;
+ }
+ vendedOptions.forEach((key, value) -> {
+ // Look the alias up in lower case, the way the S3 adapter does.
OpenDAL lower-cases
+ // every key before deserializing its config, so a vended
ACCESS_KEY_ID reaches the
+ // store as a credential either way - but left unrecognized here
it would not count as
+ // signing configuration, and the anonymous flag would be inferred
as true beside it.
+ // An unrecognized key keeps its original spelling: only the store
knows what it means.
+ String publicKey =
PUBLIC_BY_ALIAS.getOrDefault(key.toLowerCase(Locale.ROOT), key);
+ String previous = result.put(publicKey, value);
+ if (previous != null && !previous.equals(value)) {
+ throw new IllegalArgumentException(
+ "Lance namespace vended conflicting values for storage
option '"
+ + publicKey + "'");
+ }
+ });
+ return result;
+ }
+
+ @Override
+ public Map<String, String> inferStorageOptions(Map<String, String>
effectiveOptions) {
+ Map<String, String> inferred = new HashMap<>();
+ boolean hasSigningConfiguration =
hasSigningConfiguration(effectiveOptions);
+ String allowAnonymous = effectiveOptions.get(ALLOW_ANONYMOUS);
+ if (allowAnonymous != null) {
+ Boolean anonymous = parseOpenDalBoolean(allowAnonymous);
+ if (anonymous == null) {
+ throw new IllegalArgumentException("Unrecognized value for OSS
storage option '"
+ + ALLOW_ANONYMOUS + "': '" + allowAnonymous
+ + "'. Expected one of true, on, false, off");
+ }
+ if (anonymous && hasSigningConfiguration) {
+ throw new IllegalArgumentException(
+ "Conflicting OSS authentication: anonymous access is
enabled but signing "
+ + "credentials are also configured");
+ }
+ return inferred;
+ }
+ inferred.put(ALLOW_ANONYMOUS,
String.valueOf(!hasSigningConfiguration));
+ return inferred;
+ }
+
+ /**
+ * Reads a flag the way the store will. OpenDAL deserializes its config
with a boolean grammar
+ * of its own - {@code true|on} and {@code false|off}, anything else
refused - so judging a
+ * vended value by Java's rules would let {@code on} through as "not
anonymous" while the store
+ * reads it as anonymous and stops signing. Null for a value OpenDAL would
reject, so it can be
+ * refused here rather than deep inside the operator build.
+ *
+ * <p>See {@code opendal-core/src/raw/serde_util.rs}, {@code
Pair::deserialize_bool}.
+ */
+ private static Boolean parseOpenDalBoolean(String value) {
+ switch (value.toLowerCase(Locale.ROOT)) {
+ case "true":
+ case "on":
+ return Boolean.TRUE;
+ case "false":
+ case "off":
+ return Boolean.FALSE;
+ default:
+ return null;
+ }
+ }
+
+ private static boolean hasSigningConfiguration(Map<String, String>
options) {
+ return StringUtils.isNotEmpty(options.get(ACCESS_KEY_ID));
+ }
+
+ private static OSSProperties selectOss(List<StorageProperties>
storageProperties) {
+ if (storageProperties == null) {
+ return null;
+ }
+ for (StorageProperties candidate : storageProperties) {
+ if (candidate instanceof OSSProperties) {
+ return (OSSProperties) candidate;
+ }
+ }
+ return null;
+ }
+
+ private static void putIfNotEmpty(Map<String, String> target, String key,
String value) {
+ if (value != null && !value.isEmpty()) {
+ target.put(key, value);
+ }
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LancePassThroughStorageProvider.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LancePassThroughStorageProvider.java
index 78cc69ee9be..f17f7673f28 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LancePassThroughStorageProvider.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LancePassThroughStorageProvider.java
@@ -25,17 +25,16 @@ import java.util.List;
import java.util.Map;
/**
- * Every provider Lance routes somewhere other than S3 - Azure, GCS, OSS,
Tencent COS, local files,
+ * Every provider Lance routes somewhere other than S3 or OSS - Azure, GCS,
Tencent COS, local files,
* and anything Lance adds later.
*
* <p>Both halves are deliberately inert, so such a dataset is reachable only
through what its
- * namespace vends. Those options are the namespace's to name: Lance's OSS
provider reads
- * {@code access_key_id} and requires {@code endpoint}, object_store's Azure
parser reads
- * {@code endpoint} and takes {@code token} as a bearer token. Rewriting any
of that onto the S3
+ * namespace vends. Those options are the namespace's to name: object_store's
Azure parser reads
+ * {@code endpoint} and takes {@code token} as a bearer token. Rewriting
either onto the S3
* spellings would leave the dataset unreachable, which is what this class
exists to prevent.
*
- * <p>{@link #fromDorisProperties} is empty for want of a translation, not for
want of input: Doris
- * does model these - {@code OSSProperties} and {@code COSProperties} carry an
endpoint and
+ * <p>{@link #normalizeDorisStorageOptions} is empty for want of a
translation, not for want of
+ * input: Doris does model these - {@code COSProperties} carries an endpoint
and
* credentials like any other. Writing that translation means committing to a
vocabulary per
* provider with no way to exercise it here, which is how the rewriting bug
above got in, so it
* waits for a backend that can be tested against.
@@ -48,12 +47,19 @@ final class LancePassThroughStorageProvider implements
LanceStorageProvider {
}
@Override
- public Map<String, String> fromDorisProperties(List<StorageProperties>
storageProperties) {
+ public Map<String, String> normalizeDorisStorageOptions(
+ List<StorageProperties> storageProperties) {
return Collections.emptyMap();
}
@Override
- public Map<String, String> normalizeVended(Map<String, String>
vendedOptions) {
+ public Map<String, String> normalizeVendedStorageOptions(
+ Map<String, String> vendedOptions) {
return vendedOptions == null ? new HashMap<>() : new
HashMap<>(vendedOptions);
}
+
+ @Override
+ public Map<String, String> inferStorageOptions(Map<String, String>
effectiveOptions) {
+ return Collections.emptyMap();
+ }
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceS3StorageProvider.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceS3StorageProvider.java
index eb723a0201f..91d011be156 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceS3StorageProvider.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceS3StorageProvider.java
@@ -108,7 +108,8 @@ final class LanceS3StorageProvider implements
LanceStorageProvider {
}
@Override
- public Map<String, String> fromDorisProperties(List<StorageProperties>
storageProperties) {
+ public Map<String, String> normalizeDorisStorageOptions(
+ List<StorageProperties> storageProperties) {
Map<String, String> result = new HashMap<>();
AbstractS3CompatibleProperties properties =
selectS3Compatible(storageProperties);
if (properties == null) {
@@ -125,12 +126,6 @@ final class LanceS3StorageProvider implements
LanceStorageProvider {
result.put(VIRTUAL_HOSTED_STYLE,
String.valueOf(!Boolean.parseBoolean(usePathStyle)));
}
- // Lance refuses a plain-HTTP endpoint unless this is set, and Doris
configures one for
- // MinIO. It describes the endpoint just mapped, so it is derived from
the same properties.
- String endpoint = properties.getEndpoint();
- if (endpoint != null && endpoint.startsWith("http://";)) {
- result.put(ALLOW_HTTP, "true");
- }
return result;
}
@@ -166,7 +161,8 @@ final class LanceS3StorageProvider implements
LanceStorageProvider {
}
@Override
- public Map<String, String> normalizeVended(Map<String, String>
vendedOptions) {
+ public Map<String, String> normalizeVendedStorageOptions(
+ Map<String, String> vendedOptions) {
Map<String, String> result = new HashMap<>();
if (vendedOptions == null) {
return result;
@@ -185,6 +181,20 @@ final class LanceS3StorageProvider implements
LanceStorageProvider {
return result;
}
+ @Override
+ public Map<String, String> inferStorageOptions(Map<String, String>
effectiveOptions) {
+ Map<String, String> inferred = new HashMap<>();
+ if (effectiveOptions.containsKey(ALLOW_HTTP)) {
+ return inferred;
+ }
+ String endpoint = effectiveOptions.get(ENDPOINT);
+ if (endpoint != null && endpoint.startsWith("http://";)) {
+ // object_store rejects a plain-HTTP endpoint unless this client
option is enabled.
+ inferred.put(ALLOW_HTTP, "true");
+ }
+ return inferred;
+ }
+
private static void putIfNotEmpty(Map<String, String> target, String key,
String value) {
if (value != null && !value.isEmpty()) {
target.put(key, value);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceStorageOptions.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceStorageOptions.java
index 65d9ba2a8a7..c13acc8c5ea 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceStorageOptions.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceStorageOptions.java
@@ -46,12 +46,9 @@ public final class LanceStorageOptions {
* <p>Used wherever no namespace is involved: the storage a namespace
client reads itself, and
* the {@code s3()} table-valued function.
*/
- public static Map<String, String> forUri(String uri,
List<StorageProperties> storageProperties) {
- Map<String, String> result = new HashMap<>(
-
LanceStorageProvider.forDataset(uri).fromDorisProperties(storageProperties));
- result.forEach((key, value) -> rejectUntransportable(key, value,
- "Doris storage configuration"));
- return result;
+ public static Map<String, String> fromDorisStorageProperties(
+ String uri, List<StorageProperties> storageProperties) {
+ return buildStorageOptions(uri, storageProperties, null);
}
/**
@@ -63,19 +60,32 @@ public final class LanceStorageOptions {
* whichever its HashMap yields last - independently in the FE and in the
BE.
*
* <p>{@code vendedOptions} may be null or empty; a namespace that
describes a table without
- * vending storage options is ordinary, and this then degenerates to
{@link #forUri}.
+ * vending storage options is ordinary, and this then degenerates to
+ * {@link #fromDorisStorageProperties}.
*/
- public static Map<String, String> forVendedTable(String datasetUri,
+ public static Map<String, String> fromDorisAndVendedStorageOptions(String
datasetUri,
List<StorageProperties> storageProperties, Map<String, String>
vendedOptions) {
- Map<String, String> result = forUri(datasetUri, storageProperties);
- if (vendedOptions == null || vendedOptions.isEmpty()) {
- return result;
+ return buildStorageOptions(datasetUri, storageProperties,
vendedOptions);
+ }
+
+ private static Map<String, String> buildStorageOptions(String datasetUri,
+ List<StorageProperties> storageProperties, Map<String, String>
vendedOptions) {
+ LanceStorageProvider provider =
LanceStorageProvider.forDataset(datasetUri);
+ Map<String, String> result = new HashMap<>(
+ provider.normalizeDorisStorageOptions(storageProperties));
+ result.forEach((key, value) -> rejectUntransportable(key, value,
+ "Doris storage configuration"));
+ Map<String, String> normalizedVended = new HashMap<>();
+ if (vendedOptions != null && !vendedOptions.isEmpty()) {
+ vendedOptions.forEach(LanceStorageOptions::validateVendedOption);
+ // Safe to validate before normalizing: normalization only renames
a key to a provider
+ // constant or passes it through, so it cannot introduce a NUL
missed above.
+ normalizedVended =
provider.normalizeVendedStorageOptions(vendedOptions);
}
- vendedOptions.forEach(LanceStorageOptions::validateVendedOption);
- // Safe to validate the vended half before normalizing:
normalizeVended only ever renames a
- // key to one of this class's own constants or passes it through
unchanged, so it cannot
- // introduce a NUL that the check above would have missed.
-
result.putAll(LanceStorageProvider.forDataset(datasetUri).normalizeVended(vendedOptions));
+ result.putAll(normalizedVended);
+ provider.inferStorageOptions(result).forEach(result::putIfAbsent);
+ result.forEach((key, value) -> rejectUntransportable(key, value,
+ "Lance storage configuration"));
return result;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceStorageProvider.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceStorageProvider.java
index 9b980351af1..b5edfcf5587 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceStorageProvider.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceStorageProvider.java
@@ -52,26 +52,41 @@ public interface LanceStorageProvider {
* and flattens every configured storage into one namespace where two
S3-compatible ones would
* silently overwrite each other.
*
- * <p>Empty when the list holds nothing this provider can read, which is
every provider but S3
- * today - those datasets are reachable only through what a namespace
vends.
+ * <p>Empty when the list holds nothing this provider can read. Providers
without a Doris
+ * adapter are reachable only through what a namespace vends.
*/
- Map<String, String> fromDorisProperties(List<StorageProperties>
storageProperties);
+ Map<String, String> normalizeDorisStorageOptions(List<StorageProperties>
storageProperties);
/**
- * Rewrites the options a namespace vended onto the spelling {@link
#fromDorisProperties}
- * emits, so that the two cannot reach Lance as competing entries for one
config key.
+ * Rewrites the options a namespace vended onto the spelling
+ * {@link #normalizeDorisStorageOptions} emits, so that the two cannot
reach Lance as competing
+ * entries for one config key.
*
* <p>Only the options Doris itself emits are rewritten. Everything else
is passed through
* untouched: the Lance Namespace specification describes {@code
storage_options} as
* configuration "passed directly to Lance", so a client cannot assume a
vocabulary beyond the
* one it contributes to itself.
*/
- Map<String, String> normalizeVended(Map<String, String> vendedOptions);
+ Map<String, String> normalizeVendedStorageOptions(Map<String, String>
vendedOptions);
+
+ /**
+ * Derives provider options from the fully normalized and merged option
map.
+ *
+ * <p>The returned entries are defaults: an explicitly supplied option
always wins. Inference
+ * therefore runs only after catalog and namespace options have been
reconciled.
+ */
+ Map<String, String> inferStorageOptions(Map<String, String>
effectiveOptions);
/** The provider Lance will route this dataset to. */
static LanceStorageProvider forDataset(String datasetUri) {
- return S3_SCHEMES.contains(schemeOf(datasetUri))
- ? LanceS3StorageProvider.INSTANCE :
LancePassThroughStorageProvider.INSTANCE;
+ String scheme = schemeOf(datasetUri);
+ if (S3_SCHEMES.contains(scheme)) {
+ return LanceS3StorageProvider.INSTANCE;
+ }
+ if ("oss".equals(scheme)) {
+ return LanceOssStorageProvider.INSTANCE;
+ }
+ return LancePassThroughStorageProvider.INSTANCE;
}
static String schemeOf(String datasetUri) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/LanceFileSystemMetastoreProperties.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/LanceFileSystemMetastoreProperties.java
index 19aa4afef1f..0bbb1548906 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/LanceFileSystemMetastoreProperties.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/metastore/LanceFileSystemMetastoreProperties.java
@@ -17,6 +17,7 @@
package org.apache.doris.datasource.property.metastore;
+import org.apache.doris.datasource.property.storage.OSSHdfsProperties;
import org.apache.doris.foundation.property.ConnectorProperty;
import org.apache.arrow.memory.BufferAllocator;
@@ -37,7 +38,7 @@ public class LanceFileSystemMetastoreProperties extends
AbstractLanceProperties
@ConnectorProperty(
names = {WAREHOUSE},
required = false,
- description = "The local, file, or S3 warehouse containing Lance
datasets."
+ description = "The local, file, S3, or OSS warehouse containing
Lance datasets."
)
private String warehouse;
@@ -77,6 +78,7 @@ public class LanceFileSystemMetastoreProperties extends
AbstractLanceProperties
throw new IllegalArgumentException(
"Missing required property 'warehouse' for Lance
filesystem catalog");
}
+ rejectOssHdfs();
validateWarehouse(warehouse);
for (String key : origProps.keySet()) {
if (key.startsWith("lance.rest.")) {
@@ -102,9 +104,36 @@ public class LanceFileSystemMetastoreProperties extends
AbstractLanceProperties
return;
}
String scheme = uri.getScheme().toLowerCase(Locale.ROOT);
- if (!"file".equals(scheme) && !"s3".equals(scheme)) {
+ if (!"file".equals(scheme) && !"s3".equals(scheme) &&
!"oss".equals(scheme)) {
throw new IllegalArgumentException("Unsupported Lance filesystem
warehouse scheme '" + scheme
- + "'; first phase supports local/file and s3");
+ + "'; supported schemes are local/file, s3, and oss");
+ }
+ // An object-store root names its bucket in the authority. Lance reads
that authority as the
+ // bucket and fails deep inside the store when it is absent, so reject
the no-authority form
+ // here where the message can still name the property.
+ if (("s3".equals(scheme) || "oss".equals(scheme)) &&
StringUtils.isBlank(uri.getAuthority())) {
+ throw new IllegalArgumentException(
+ "Lance " + scheme + " warehouse must name a bucket, as in
" + scheme
+ + "://bucket/path, but was: " + warehouse);
}
}
+
+ /**
+ * Doris routes an OSS-HDFS configuration to {@code OSSHdfsProperties},
which the Lance OSS
+ * provider cannot read - it accepts only {@code OSSProperties} - so the
namespace would be
+ * handed no endpoint and no credentials and could not open at all.
+ *
+ * <p>Checks every property rather than only the warehouse: {@code
OSSHdfsProperties.guessIsMe}
+ * selects on the endpoint, so {@code oss://bucket/path} with an {@code
oss.endpoint} ending in
+ * the OSS-HDFS suffix routes there just the same, with a clean-looking
warehouse.
+ */
+ private void rejectOssHdfs() {
+ if (OSSHdfsProperties.guessIsMe(origProps)) {
+ throw new IllegalArgumentException(
+ "OSS-HDFS is not supported by the Lance catalog. Doris
reads this form through "
+ + "its HDFS-compatible properties, which carry no
Lance OSS storage "
+ + "options.");
+ }
+ }
+
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java
index f7e0e86c5b0..99da0093513 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/TVFScanNode.java
@@ -133,7 +133,7 @@ public class TVFScanNode extends FileQueryScanNode {
if (tableValuedFunction.isLanceFormat()) {
// lance-c opens the dataset itself and needs the options in
Lance's own vocabulary.
// Set at ScanNode level so credentials are not serialized once
per fragment split.
- Map<String, String> lanceStorageOptions =
LanceStorageOptions.forUri(
+ Map<String, String> lanceStorageOptions =
LanceStorageOptions.fromDorisStorageProperties(
tableValuedFunction.getFilePath(),
Collections.singletonList(tableValuedFunction.getStorageProperties()));
if (!lanceStorageOptions.isEmpty()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java
index 8955deba76f..120156b54c7 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java
@@ -539,7 +539,7 @@ public abstract class ExternalFileTableValuedFunction
extends TableValuedFunctio
fileScanRangeParams.setProperties(beProperties);
if (fileFormatProperties.getFileFormatType() ==
TFileFormatType.FORMAT_LANCE) {
// lance-c opens the dataset itself and needs the options in
Lance's own vocabulary.
- Map<String, String> lanceStorageOptions =
LanceStorageOptions.forUri(
+ Map<String, String> lanceStorageOptions =
LanceStorageOptions.fromDorisStorageProperties(
filePath, Collections.singletonList(storageProperties));
if (!lanceStorageOptions.isEmpty()) {
fileScanRangeParams.setLanceScanParams(
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceFilesystemCatalogTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceFilesystemCatalogTest.java
index df008abe564..95acd580b22 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceFilesystemCatalogTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceFilesystemCatalogTest.java
@@ -146,6 +146,9 @@ public class LanceFilesystemCatalogTest {
String accessKey = "sentinel-access-key";
String secretKey = "sentinel-secret-key";
String sessionToken = "sentinel-session-token";
+ String ossAccessKey = "sentinel-oss-access-key";
+ String ossSecretKey = "sentinel-oss-secret-key";
+ String ossSecurityToken = "sentinel-oss-security-token";
String datasetUri =
"s3://sentinel-user:sentinel-password@bucket/private/table.lance";
Map<String, String> catalogProperties = new HashMap<>();
@@ -158,9 +161,14 @@ public class LanceFilesystemCatalogTest {
runtimeStorageOptions.put("aws_access_key_id", accessKey);
runtimeStorageOptions.put("aws_secret_access_key", secretKey);
runtimeStorageOptions.put("aws_session_token", sessionToken);
+ runtimeStorageOptions.put("oss_access_key_id", ossAccessKey);
+ runtimeStorageOptions.put("oss_secret_access_key", ossSecretKey);
+ runtimeStorageOptions.put("oss_security_token", ossSecurityToken);
String providerMessage = "provider failure\nuri=" + datasetUri
+ " bearer=" + bearerToken + " api-key=" + apiKey
- + " access=" + accessKey + " secret=" + secretKey + "
session=" + sessionToken;
+ + " access=" + accessKey + " secret=" + secretKey + "
session=" + sessionToken
+ + " oss-access=" + ossAccessKey + " oss-secret=" + ossSecretKey
+ + " oss-token=" + ossSecurityToken;
RuntimeException providerFailure = new
RuntimeException(providerMessage);
RuntimeException exposed = catalog.indexMetadataLoadFailure(
@@ -169,7 +177,7 @@ public class LanceFilesystemCatalogTest {
exposed.printStackTrace(new PrintWriter(stackTrace));
for (String sentinel : Arrays.asList(bearerToken, apiKey, accessKey,
secretKey,
- sessionToken, datasetUri)) {
+ sessionToken, ossAccessKey, ossSecretKey, ossSecurityToken,
datasetUri)) {
Assert.assertFalse(exposed.getMessage().contains(sentinel));
Assert.assertFalse(exposed.getCause().getMessage().contains(sentinel));
Assert.assertFalse(stackTrace.toString().contains(sentinel));
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceStorageOptionsTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceStorageOptionsTest.java
index 9e3ca4c5f4a..be25f115ae9 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceStorageOptionsTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceStorageOptionsTest.java
@@ -32,6 +32,7 @@ import java.util.TreeSet;
public class LanceStorageOptionsTest {
private static final String S3_URI = "s3://warehouse/table.lance";
+ private static final String OSS_URI = "oss://warehouse/table.lance";
/**
* Parsed by Doris exactly as a real catalog would, so these fixtures also
have to satisfy its
@@ -60,13 +61,23 @@ public class LanceStorageOptionsTest {
return createAll(minioProperties());
}
+ private static Map<String, String> ossProperties() {
+ Map<String, String> properties = new HashMap<>();
+ properties.put("fs.oss.support", "true");
+ properties.put("oss.endpoint", "https://oss-cn-hangzhou.aliyuncs.com";);
+ properties.put("oss.region", "cn-hangzhou");
+ properties.put("oss.access_key", "oss-ak");
+ properties.put("oss.secret_key", "oss-sk");
+ return properties;
+ }
+
@Test
public void testCatalogPropertiesMapToTheCanonicalS3Spelling() {
Map<String, String> properties = minioProperties();
properties.put("s3.session_token", "token");
Map<String, String> options =
- LanceStorageOptions.forUri(S3_URI, createAll(properties));
+ LanceStorageOptions.fromDorisStorageProperties(S3_URI,
createAll(properties));
Assertions.assertEquals("ak", options.get("aws_access_key_id"));
Assertions.assertEquals("sk", options.get("aws_secret_access_key"));
Assertions.assertEquals("token", options.get("aws_session_token"));
@@ -91,7 +102,7 @@ public class LanceStorageOptionsTest {
properties.put("s3.region", "us-east-1");
Map<String, String> options =
- LanceStorageOptions.forUri(S3_URI, createAll(properties));
+ LanceStorageOptions.fromDorisStorageProperties(S3_URI,
createAll(properties));
Assertions.assertNull(options.get("aws_access_key_id"));
Assertions.assertNull(options.get("aws_secret_access_key"));
Assertions.assertNull(options.get("aws_session_token"));
@@ -100,6 +111,103 @@ public class LanceStorageOptionsTest {
Assertions.assertEquals("https://s3.amazonaws.com";,
options.get("aws_endpoint"));
}
+ @Test
+ public void testOssCatalogPropertiesMapToThePublicSpelling() {
+ Map<String, String> properties = ossProperties();
+ properties.put("oss.session_token", "oss-token");
+
+ Map<String, String> options =
LanceStorageOptions.fromDorisStorageProperties(
+ OSS_URI, createAll(properties));
+
+ Assertions.assertEquals("https://oss-cn-hangzhou.aliyuncs.com";,
+ options.get("oss_endpoint"));
+ Assertions.assertEquals("oss-ak", options.get("oss_access_key_id"));
+ Assertions.assertEquals("oss-sk",
options.get("oss_secret_access_key"));
+ Assertions.assertEquals("cn-hangzhou", options.get("oss_region"));
+ Assertions.assertEquals("oss-token",
options.get("oss_security_token"));
+ // Stated rather than implied, so the host environment cannot decide
it; see
+ // testOssVirtualHostAddressingIsStatedExplicitly.
+ Assertions.assertEquals("virtual", options.get("addressing_style"));
+ Assertions.assertEquals(
+ new TreeSet<>(Arrays.asList("oss_endpoint",
"oss_access_key_id",
+ "oss_secret_access_key", "oss_region",
"oss_security_token",
+ "addressing_style", "allow_anonymous")),
+ new TreeSet<>(options.keySet()));
+ Assertions.assertEquals("false", options.get("allow_anonymous"));
+ }
+
+ @Test
+ public void testOssAnonymousAccessEmitsNoCredentials() {
+ Map<String, String> properties = ossProperties();
+ properties.remove("oss.access_key");
+ properties.remove("oss.secret_key");
+
+ Map<String, String> options =
LanceStorageOptions.fromDorisStorageProperties(
+ OSS_URI, createAll(properties));
+
+ Assertions.assertEquals("https://oss-cn-hangzhou.aliyuncs.com";,
+ options.get("oss_endpoint"));
+ Assertions.assertEquals("cn-hangzhou", options.get("oss_region"));
+ Assertions.assertNull(options.get("oss_access_key_id"));
+ Assertions.assertNull(options.get("oss_secret_access_key"));
+ Assertions.assertNull(options.get("oss_security_token"));
+ }
+
+ @Test
+ public void testVendedOssOptionsSupersedeTheCatalog() {
+ Map<String, String> properties = ossProperties();
+ properties.put("oss.session_token", "static-token");
+ Map<String, String> vended = new HashMap<>();
+ vended.put("endpoint", "https://oss-cn-shanghai.aliyuncs.com";);
+ vended.put("access_key_id", "vended-ak");
+ vended.put("access_key_secret", "vended-sk");
+ vended.put("region", "cn-shanghai");
+ vended.put("security_token", "vended-token");
+
+ Map<String, String> merged =
LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, createAll(properties), vended);
+
+ Assertions.assertEquals("https://oss-cn-shanghai.aliyuncs.com";,
+ merged.get("oss_endpoint"));
+ Assertions.assertEquals("vended-ak", merged.get("oss_access_key_id"));
+ Assertions.assertEquals("vended-sk",
merged.get("oss_secret_access_key"));
+ Assertions.assertEquals("cn-shanghai", merged.get("oss_region"));
+ Assertions.assertEquals("vended-token",
merged.get("oss_security_token"));
+ for (String alias : vended.keySet()) {
+ Assertions.assertNull(merged.get(alias), alias + " must not
survive beside its twin");
+ }
+ }
+
+ @Test
+ public void testOssAliasesCollapseAndConflictsAreRejected() {
+ // Carries a whole credential: a lone secret is a one-sided pair,
which reconciliation
+ // now rejects on its own account, and that is not what this case is
about.
+ Map<String, String> agreeing = new HashMap<>();
+ agreeing.put("access_key_id", "ak");
+ agreeing.put("access_key_secret", "same");
+ agreeing.put("oss_secret_access_key", "same");
+ Assertions.assertEquals("same",
LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, Collections.emptyList(),
agreeing).get("oss_secret_access_key"));
+
+ Map<String, String> conflicting = new HashMap<>();
+ conflicting.put("endpoint", "https://one.example.com";);
+ conflicting.put("oss_endpoint", "https://another.example.com";);
+ Assertions.assertThrows(IllegalArgumentException.class, () ->
LanceStorageOptions
+ .fromDorisAndVendedStorageOptions(OSS_URI,
Collections.emptyList(), conflicting));
+ }
+
+ @Test
+ public void testUnknownVendedOssOptionsPassThrough() {
+ Map<String, String> vended = new HashMap<>();
+ vended.put("unknown_provider_option", "value");
+
+ Map<String, String> merged =
LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, Collections.emptyList(), vended);
+
+ Assertions.assertEquals("value",
merged.get("unknown_provider_option"));
+ Assertions.assertEquals("true", merged.get("allow_anonymous"));
+ }
+
/**
* The case this exists for: a catalog with static credentials whose
namespace also vends them,
* spelled the way real servers spell them. Both must land on one key with
the namespace's
@@ -116,7 +224,7 @@ public class LanceStorageOptionsTest {
vended.put("virtual_hosted_style_request", "true");
Map<String, String> merged =
- LanceStorageOptions.forVendedTable(S3_URI, minioCatalog(),
vended);
+ LanceStorageOptions.fromDorisAndVendedStorageOptions(S3_URI,
minioCatalog(), vended);
Assertions.assertEquals("vended-ak", merged.get("aws_access_key_id"));
Assertions.assertEquals("vended-sk",
merged.get("aws_secret_access_key"));
@@ -139,7 +247,7 @@ public class LanceStorageOptionsTest {
vended.put(alias, "http://127.0.0.1:9000";);
Map<String, String> merged =
- LanceStorageOptions.forVendedTable(S3_URI, minioCatalog(),
vended);
+
LanceStorageOptions.fromDorisAndVendedStorageOptions(S3_URI, minioCatalog(),
vended);
long endpoints = merged.keySet().stream().filter(k ->
k.contains("endpoint")).count();
Assertions.assertEquals(1, endpoints, alias + " left a competing
entry");
Assertions.assertEquals("http://127.0.0.1:9000";,
merged.get("aws_endpoint"),
@@ -147,6 +255,37 @@ public class LanceStorageOptionsTest {
}
}
+ @Test
+ public void testS3AllowHttpIsInferredAfterVendedEndpointWins() {
+ Map<String, String> httpsEndpoint = Collections.singletonMap(
+ "endpoint", "https://s3.example.com";);
+ Map<String, String> httpsOptions = LanceStorageOptions
+ .fromDorisAndVendedStorageOptions(S3_URI, minioCatalog(),
httpsEndpoint);
+ Assertions.assertEquals("https://s3.example.com";,
httpsOptions.get("aws_endpoint"));
+ Assertions.assertNull(httpsOptions.get("allow_http"));
+
+ Map<String, String> explicitFalse = new HashMap<>();
+ explicitFalse.put("endpoint", "http://127.0.0.1:9000";);
+ explicitFalse.put("allow_http", "false");
+ Map<String, String> explicitOptions = LanceStorageOptions
+ .fromDorisAndVendedStorageOptions(S3_URI, minioCatalog(),
explicitFalse);
+ Assertions.assertEquals("false", explicitOptions.get("allow_http"));
+ }
+
+ @Test
+ public void testNormalizationDoesNotInferProviderOptions() {
+ Map<String, String> s3 = LanceS3StorageProvider.INSTANCE
+ .normalizeDorisStorageOptions(minioCatalog());
+ Assertions.assertEquals("http://minio:9000";, s3.get("aws_endpoint"));
+ Assertions.assertFalse(s3.containsKey("allow_http"));
+
+ Map<String, String> oss = LanceOssStorageProvider.INSTANCE
+ .normalizeDorisStorageOptions(createAll(ossProperties()));
+ Assertions.assertEquals("oss-ak", oss.get("oss_access_key_id"));
+ Assertions.assertFalse(oss.containsKey("skip_signature"));
+ Assertions.assertFalse(oss.containsKey("allow_anonymous"));
+ }
+
/**
* {@code token} means an S3 session token to object_store's S3 parser and
a bearer token to
* its Azure one. Knowing the provider is what makes it resolvable at all.
@@ -160,11 +299,11 @@ public class LanceStorageOptionsTest {
Map<String, String> vended = new HashMap<>();
vended.put("token", "vended-token");
- Map<String, String> onS3 = LanceStorageOptions.forVendedTable(S3_URI,
catalog, vended);
+ Map<String, String> onS3 =
LanceStorageOptions.fromDorisAndVendedStorageOptions(S3_URI, catalog, vended);
Assertions.assertEquals("vended-token", onS3.get("aws_session_token"));
Assertions.assertNull(onS3.get("token"));
- Map<String, String> onAzure = LanceStorageOptions.forVendedTable(
+ Map<String, String> onAzure =
LanceStorageOptions.fromDorisAndVendedStorageOptions(
"az://container/table.lance", catalog, vended);
Assertions.assertEquals("vended-token", onAzure.get("token"));
Assertions.assertNull(onAzure.get("aws_session_token"));
@@ -172,9 +311,8 @@ public class LanceStorageOptionsTest {
/**
* The regression that motivated all of this: object_store's Azure parser
reads
- * {@code endpoint} but not {@code aws_endpoint}, and Lance's OSS provider
requires
- * {@code endpoint} and reads {@code access_key_id}. Rewriting those onto
the S3 spellings
- * leaves the dataset unreachable, so a non-S3 dataset must come through
untouched.
+ * {@code endpoint} but not {@code aws_endpoint}. Rewriting it onto the S3
spelling leaves the
+ * dataset unreachable, so a provider with no Doris adapter must come
through untouched.
*/
@Test
public void testNonS3DatasetsAreNeverRewritten() {
@@ -184,10 +322,10 @@ public class LanceStorageOptionsTest {
vended.put("secret_access_key", "vended-sk");
for (String uri : new String[] {"az://container/table.lance",
"abfss://fs@acct/table",
- "oss://bucket/table.lance", "gs://bucket/table.lance",
"cos://bucket/table",
+ "gs://bucket/table.lance", "cos://bucket/table",
"file:///tmp/table.lance"}) {
Map<String, String> merged =
- LanceStorageOptions.forVendedTable(uri, minioCatalog(),
vended);
+ LanceStorageOptions.fromDorisAndVendedStorageOptions(uri,
minioCatalog(), vended);
Assertions.assertEquals(vended, merged,
uri + " must reach Lance exactly as the namespace wrote
it");
}
@@ -202,17 +340,18 @@ public class LanceStorageOptionsTest {
public void testCatalogWithoutAnS3UrlGetsNoS3Options() {
for (String uri : new String[] {"file:///warehouse/lance", "", null}) {
Assertions.assertTrue(
- LanceStorageOptions.forUri(uri, minioCatalog()).isEmpty(),
+ LanceStorageOptions.fromDorisStorageProperties(uri,
minioCatalog()).isEmpty(),
"expected no options for " + uri);
}
}
- /** Doris has no Lance vocabulary for a non-S3 provider yet, so it
contributes none. */
+ /** A provider never receives another provider's static configuration. */
@Test
public void testCatalogPropertiesAreNotAppliedToANonS3Dataset() {
- Map<String, String> merged = LanceStorageOptions.forUri(
+ Map<String, String> merged =
LanceStorageOptions.fromDorisStorageProperties(
"oss://bucket/table.lance", minioCatalog());
- Assertions.assertTrue(merged.isEmpty(), "S3 credentials must not leak
onto another provider");
+ Assertions.assertEquals(Collections.singletonMap("allow_anonymous",
"true"), merged,
+ "S3 credentials must not leak onto another provider");
}
/**
@@ -225,7 +364,7 @@ public class LanceStorageOptionsTest {
Assertions.assertTrue(catalog.size() > 1,
"expected Doris to add its default non-S3 entry ahead of the
S3 one");
Assertions.assertEquals("ak",
- LanceStorageOptions.forUri(S3_URI,
catalog).get("aws_access_key_id"));
+ LanceStorageOptions.fromDorisStorageProperties(S3_URI,
catalog).get("aws_access_key_id"));
}
@Test
@@ -238,18 +377,18 @@ public class LanceStorageOptionsTest {
vended.put("deliberately_empty", "");
Map<String, String> merged =
- LanceStorageOptions.forVendedTable(S3_URI,
Collections.emptyList(), vended);
+ LanceStorageOptions.fromDorisAndVendedStorageOptions(S3_URI,
Collections.emptyList(), vended);
Assertions.assertEquals(vended, merged);
}
@Test
public void testAbsentVendedOptionsLeaveCatalogOptionsIntact() {
Map<String, String> catalogOptions =
- LanceStorageOptions.forUri(S3_URI, minioCatalog());
+ LanceStorageOptions.fromDorisStorageProperties(S3_URI,
minioCatalog());
Assertions.assertEquals(catalogOptions,
- LanceStorageOptions.forUri(S3_URI, minioCatalog()));
+ LanceStorageOptions.fromDorisStorageProperties(S3_URI,
minioCatalog()));
Assertions.assertEquals(catalogOptions,
- LanceStorageOptions.forVendedTable(S3_URI, minioCatalog(), new
HashMap<>()));
+ LanceStorageOptions.fromDorisAndVendedStorageOptions(S3_URI,
minioCatalog(), new HashMap<>()));
}
/** A namespace contradicting itself is not something to resolve by coin
toss. */
@@ -259,14 +398,14 @@ public class LanceStorageOptionsTest {
vended.put("access_key_id", "one");
vended.put("aws_access_key_id", "another");
Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceStorageOptions.forVendedTable(S3_URI,
minioCatalog(), vended));
+ () ->
LanceStorageOptions.fromDorisAndVendedStorageOptions(S3_URI, minioCatalog(),
vended));
// Agreeing on the value is not a conflict.
Map<String, String> agreeing = new HashMap<>();
agreeing.put("access_key_id", "same");
agreeing.put("aws_access_key_id", "same");
Assertions.assertEquals("same", LanceStorageOptions
- .forVendedTable(S3_URI, minioCatalog(),
agreeing).get("aws_access_key_id"));
+ .fromDorisAndVendedStorageOptions(S3_URI, minioCatalog(),
agreeing).get("aws_access_key_id"));
}
/**
@@ -279,7 +418,7 @@ public class LanceStorageOptionsTest {
properties.put("s3.access_key", "ak\0ignored");
IllegalArgumentException thrown =
Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceStorageOptions.forUri(S3_URI,
createAll(properties)));
+ () -> LanceStorageOptions.fromDorisStorageProperties(S3_URI,
createAll(properties)));
// The message has to say which side to fix - the catalog, not the
namespace.
Assertions.assertTrue(thrown.getMessage().contains("Doris storage
configuration"),
"unexpected message: " + thrown.getMessage());
@@ -294,21 +433,332 @@ public class LanceStorageOptionsTest {
Map<String, String> withNulKey = new HashMap<>();
withNulKey.put("bucket\0ignored", "other-bucket");
Assertions.assertThrows(IllegalArgumentException.class, () ->
LanceStorageOptions
- .forVendedTable(S3_URI, Collections.emptyList(), withNulKey));
+ .fromDorisAndVendedStorageOptions(S3_URI,
Collections.emptyList(), withNulKey));
Map<String, String> withNulValue = new HashMap<>();
withNulValue.put("aws_region", "us-east-1\0ignored");
Assertions.assertThrows(IllegalArgumentException.class, () ->
LanceStorageOptions
- .forVendedTable(S3_URI, Collections.emptyList(),
withNulValue));
+ .fromDorisAndVendedStorageOptions(S3_URI,
Collections.emptyList(), withNulValue));
Map<String, String> withNullValue = new HashMap<>();
withNullValue.put("aws_region", null);
Assertions.assertThrows(IllegalArgumentException.class, () ->
LanceStorageOptions
- .forVendedTable(S3_URI, Collections.emptyList(),
withNullValue));
+ .fromDorisAndVendedStorageOptions(S3_URI,
Collections.emptyList(), withNullValue));
Map<String, String> withNullKey = new HashMap<>();
withNullKey.put(null, "value");
Assertions.assertThrows(IllegalArgumentException.class, () ->
LanceStorageOptions
- .forVendedTable(S3_URI, Collections.emptyList(), withNullKey));
+ .fromDorisAndVendedStorageOptions(S3_URI,
Collections.emptyList(), withNullKey));
+ }
+
+ /**
+ * A catalog can be absent entirely - a Lance table reached purely through
vended options has no
+ * Doris storage properties behind it. The provider has to answer with no
options rather than
+ * fail, so that {@link
LanceStorageOptions#fromDorisAndVendedStorageOptions} still has a map to merge
onto.
+ */
+ @Test
+ public void testOssDatasetWithoutACatalogInfersAnonymousAccess() {
+ Assertions.assertEquals(Collections.singletonMap("allow_anonymous",
"true"),
+ LanceStorageOptions.fromDorisStorageProperties(OSS_URI, null));
+ }
+
+ /**
+ * {@link LanceStorageOptions#fromDorisAndVendedStorageOptions} screens
out a null vended map before it reaches a
+ * provider, but the interface still admits one and the sibling providers
all tolerate it. Pin
+ * that down directly, since no public entry point can reach it.
+ */
+ @Test
+ public void testOssNormalizeVendedToleratesNoVendedOptions() {
+ Assertions.assertTrue(
+
LanceOssStorageProvider.INSTANCE.normalizeVendedStorageOptions(null).isEmpty());
+ }
+
+ /**
+ * Doris reads a blank key pair as anonymous access. OpenDAL only skips
signing when it is told
+ * to, so the Lance-native flag has to be emitted rather than merely
leaving credentials out.
+ */
+ @Test
+ public void testOssAnonymousModeIsForwardedToOpenDal() {
+ Map<String, String> properties = ossProperties();
+ properties.remove("oss.access_key");
+ properties.remove("oss.secret_key");
+
+ Map<String, String> options =
LanceStorageOptions.fromDorisStorageProperties(OSS_URI, createAll(properties));
+
+ Assertions.assertEquals("true", options.get("allow_anonymous"));
+ Assertions.assertNull(options.get("oss_access_key_id"));
+ Assertions.assertNull(options.get("oss_secret_access_key"));
+ }
+
+ @Test
+ public void testOssCredentialedModeRequiresSigning() {
+ Map<String, String> options =
LanceStorageOptions.fromDorisStorageProperties(
+ OSS_URI, createAll(ossProperties()));
+
+ Assertions.assertEquals("false", options.get("allow_anonymous"));
+ }
+
+ /**
+ * Doris models path-style addressing as a property with a default, so
both answers are stated
+ * rather than only the non-default one - see
+ * {@link #testOssVirtualHostAddressingIsStatedExplicitly} for why absence
is not neutral here.
+ */
+ @Test
+ public void testOssPathStyleAddressingIsForwarded() {
+ Map<String, String> properties = ossProperties();
+ properties.put("oss.use_path_style", "true");
+
+ Map<String, String> options =
LanceStorageOptions.fromDorisStorageProperties(OSS_URI, createAll(properties));
+
+ Assertions.assertEquals("path", options.get("addressing_style"));
+ }
+
+ /**
+ * Lance snapshots the host's {@code OSS_} and {@code AWS_} environment
variables into the same
+ * config map before storage options are applied, so the default has to be
stated rather than
+ * left implicit - otherwise an exported OSS_ADDRESSING_STYLE would
outrank the catalog.
+ */
+ @Test
+ public void testOssVirtualHostAddressingIsStatedExplicitly() {
+ Map<String, String> options =
LanceStorageOptions.fromDorisStorageProperties(
+ OSS_URI, createAll(ossProperties()));
+
+ Assertions.assertEquals("virtual", options.get("addressing_style"));
+ }
+
+ /**
+ * Doris accepts a whitespace-only key pair as "both set", so the emitted
credentials and the
+ * signing choice have to agree: whatever this provider treats as a usable
credential must
+ * suppress anonymous access, or OpenDAL sends the request unsigned.
+ */
+ @Test
+ public void testWhitespaceOssCredentialsAreNotAlsoCalledAnonymous() {
+ Map<String, String> properties = ossProperties();
+ properties.put("oss.access_key", " ");
+ properties.put("oss.secret_key", " ");
+
+ Map<String, String> options =
LanceStorageOptions.fromDorisStorageProperties(OSS_URI, createAll(properties));
+
+ Assertions.assertNotEquals(
+ options.containsKey("oss_access_key_id"),
+ Boolean.parseBoolean(options.get("allow_anonymous")),
+ "a credential and allow_anonymous=true must never both be
emitted");
+ }
+
+ /**
+ * The case that makes this matter: an anonymous catalog whose namespace
vends real credentials.
+ * OpenDAL's OSS signer returns the request unsigned whenever anonymous
access is enabled, no
+ * matter what credentials sit beside it. Leaving the flag on would
silently unsign every FE
+ * metadata read and BE scan that the vended credentials were supposed to
authorize.
+ */
+ @Test
+ public void testVendedOssCredentialsClearStaticAnonymousMode() {
+ Map<String, String> properties = ossProperties();
+ properties.remove("oss.access_key");
+ properties.remove("oss.secret_key");
+ Assertions.assertEquals("true",
+ LanceStorageOptions.fromDorisStorageProperties(OSS_URI,
createAll(properties))
+ .get("allow_anonymous"),
+ "precondition: the static half alone is anonymous");
+
+ Map<String, String> vended = new HashMap<>();
+ vended.put("access_key_id", "vended-ak");
+ vended.put("access_key_secret", "vended-sk");
+
+ Map<String, String> merged =
LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, createAll(properties), vended);
+
+ Assertions.assertEquals("false", merged.get("allow_anonymous"));
+ Assertions.assertEquals("vended-ak", merged.get("oss_access_key_id"));
+ Assertions.assertEquals("vended-sk",
merged.get("oss_secret_access_key"));
+ }
+
+ /**
+ * A namespace vending a whole credential has all of it carried through,
token included, and the
+ * signing flag inferred from it. Options merge key by key, so a namespace
is expected to vend
+ * the credential it wants used in full - what the catalog holds is not
combined with it.
+ */
+ @Test
+ public void testVendedOssTokenSurvivesWithItsOwnPair() {
+ Map<String, String> properties = ossProperties();
+ properties.put("oss.session_token", "static-token");
+ Map<String, String> vended = new HashMap<>();
+ vended.put("access_key_id", "vended-ak");
+ vended.put("access_key_secret", "vended-sk");
+ vended.put("security_token", "vended-token");
+
+ Map<String, String> merged =
LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, createAll(properties), vended);
+
+ Assertions.assertEquals("vended-token",
merged.get("oss_security_token"));
+ Assertions.assertEquals("false", merged.get("allow_anonymous"));
+ }
+
+ /**
+ * A namespace asking for anonymous access has just described this table,
which outranks a
+ * credential the catalog holds for something else. Signing with it would
403 on a bucket the
+ * catalog's account cannot read.
+ */
+ @Test
+ public void testVendedAnonymousModeConflictsWithStaticOssCredentials() {
+ Map<String, String> vended = new HashMap<>();
+ vended.put("allow_anonymous", "true");
+
+ // The catalog's credentials are not cleared to make room for it: a
namespace asking for
+ // anonymous access to a catalog that holds a credential is a
contradiction, and saying so
+ // beats silently picking one of the two.
+ IllegalArgumentException thrown = Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () -> LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, createAll(ossProperties()), vended));
+ Assertions.assertTrue(thrown.getMessage().contains("Conflicting OSS
authentication"),
+ "unexpected message: " + thrown.getMessage());
+ }
+
+ @Test
+ public void testOssAnonymousAliasesCollapseAndConflict() {
+ Map<String, String> agreeing = new HashMap<>();
+ agreeing.put("allow_anonymous", "true");
+ agreeing.put("skip_signature", "true");
+ Assertions.assertEquals("true", LanceStorageOptions
+ .fromDorisAndVendedStorageOptions(OSS_URI,
Collections.emptyList(), agreeing)
+ .get("allow_anonymous"));
+
+ Map<String, String> conflicting = new HashMap<>();
+ conflicting.put("allow_anonymous", "true");
+ conflicting.put("skip_signature", "false");
+ Assertions.assertThrows(IllegalArgumentException.class, () ->
LanceStorageOptions
+ .fromDorisAndVendedStorageOptions(
+ OSS_URI, Collections.emptyList(), conflicting));
+ }
+
+ @Test
+ public void testOssAnonymousConflictsWithSigningConfiguration() {
+ Map<String, String> vended = new HashMap<>();
+ vended.put("skip_signature", "true");
+ vended.put("access_key_id", "ak");
+ vended.put("access_key_secret", "sk");
+
+ IllegalArgumentException thrown = Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () -> LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, Collections.emptyList(), vended));
+ Assertions.assertTrue(thrown.getMessage().contains("Conflicting OSS
authentication"),
+ "unexpected message: " + thrown.getMessage());
+ }
+
+ @Test
+ public void testOssExplicitSigningChoiceWinsAndInferenceIsIdempotent() {
+ Map<String, String> vended =
Collections.singletonMap("skip_signature", "false");
+ Map<String, String> explicit =
LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, Collections.emptyList(), vended);
+ Assertions.assertEquals("false", explicit.get("allow_anonymous"));
+ Assertions.assertFalse(explicit.containsKey("skip_signature"));
+
+ Map<String, String> inferred =
LanceStorageOptions.fromDorisStorageProperties(
+ OSS_URI, createAll(ossProperties()));
+ Assertions.assertEquals(inferred,
LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, Collections.emptyList(), inferred));
+ }
+
+ /**
+ * Blank is not a credential. OpenDAL would otherwise build a signer from
empty strings and
+ * sign every request with them, while nothing says the access is
anonymous.
+ */
+ @Test
+ public void testVendedEmptyOssCredentialsFallBackToAnonymous() {
+ Map<String, String> properties = ossProperties();
+ properties.put("oss.session_token", "static-token");
+ Map<String, String> vended = new HashMap<>();
+ vended.put("access_key_id", "");
+ vended.put("access_key_secret", "");
+
+ Map<String, String> merged =
LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, createAll(properties), vended);
+
+ // The blanks themselves survive - the merge overlays key by key and
does not judge values.
+ // What matters is that they are not mistaken for a credential:
inference reads them as
+ // absent, so the request is marked anonymous rather than signed with
empty strings.
+ Assertions.assertEquals("true", merged.get("allow_anonymous"));
+ Assertions.assertEquals("", merged.get("oss_access_key_id"));
+ }
+
+ /**
+ * Absence is not neutral: Lance and OpenDAL may otherwise resolve
authentication from their
+ * environment. Doris states whether the final normalized configuration
should be signed.
+ */
+ @Test
+ public void testOssAnonymousModeIsStatedEitherWay() {
+ Assertions.assertEquals("false",
+ LanceStorageOptions.fromDorisStorageProperties(OSS_URI,
createAll(ossProperties()))
+ .get("allow_anonymous"));
+
+ Map<String, String> anonymous = ossProperties();
+ anonymous.remove("oss.access_key");
+ anonymous.remove("oss.secret_key");
+ Assertions.assertEquals("true",
+ LanceStorageOptions.fromDorisStorageProperties(OSS_URI,
createAll(anonymous))
+ .get("allow_anonymous"));
+ }
+
+ /**
+ * OpenDAL lower-cases every config key before deserializing, so a
credential vended in upper
+ * case still reaches the store. Left unrecognized here it would not count
as signing
+ * configuration, and the anonymous flag would be inferred as true beside
it - the store would
+ * then load the credential and skip signing anyway, sending every request
unsigned.
+ */
+ @Test
+ public void testVendedOssOptionsAreRecognizedRegardlessOfCase() {
+ Map<String, String> vended = new HashMap<>();
+ vended.put("ACCESS_KEY_ID", "vended-ak");
+ vended.put("Access_Key_Secret", "vended-sk");
+
+ Map<String, String> merged =
LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, Collections.emptyList(), vended);
+
+ Assertions.assertEquals("vended-ak", merged.get("oss_access_key_id"));
+ Assertions.assertEquals("vended-sk",
merged.get("oss_secret_access_key"));
+ Assertions.assertEquals("false", merged.get("allow_anonymous"));
+ }
+
+ /**
+ * The signing flag is read with OpenDAL's boolean grammar, not Java's.
Judging {@code on} by
+ * Boolean.parseBoolean would call it "not anonymous" and let it through
beside a credential,
+ * while the store reads it as anonymous and stops signing.
+ */
+ @Test
+ public void testVendedOssAnonymousFlagUsesTheStoresBooleanGrammar() {
+ for (String spelling : new String[] {"true", "on", "ON"}) {
+ Map<String, String> vended = new HashMap<>();
+ vended.put("allow_anonymous", spelling);
+ IllegalArgumentException thrown = Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () -> LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, createAll(ossProperties()), vended),
+ "'" + spelling + "' should conflict with configured
credentials");
+ Assertions.assertTrue(thrown.getMessage().contains("Conflicting
OSS authentication"),
+ "unexpected message: " + thrown.getMessage());
+ }
+ // off/false are the store's own spellings for "sign", and agree with
a credential
+ for (String spelling : new String[] {"false", "off"}) {
+ Map<String, String> vended = new HashMap<>();
+ vended.put("allow_anonymous", spelling);
+ Assertions.assertEquals(spelling,
LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, createAll(ossProperties()),
vended).get("allow_anonymous"));
+ }
+ }
+
+ /** A value OpenDAL cannot parse is refused here, not several layers down
in the operator build. */
+ @Test
+ public void testVendedOssAnonymousFlagRejectsAnUnparsableValue() {
+ Map<String, String> vended = new HashMap<>();
+ vended.put("allow_anonymous", "yes");
+
+ IllegalArgumentException thrown = Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () -> LanceStorageOptions.fromDorisAndVendedStorageOptions(
+ OSS_URI, Collections.emptyList(), vended));
+ Assertions.assertTrue(thrown.getMessage().contains("Unrecognized
value"),
+ "unexpected message: " + thrown.getMessage());
}
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/LancePropertiesTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/LancePropertiesTest.java
index 1702d583f4f..b079060ead2 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/LancePropertiesTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/metastore/LancePropertiesTest.java
@@ -42,6 +42,21 @@ public class LancePropertiesTest {
lanceProperties.getRootDatabase());
}
+ @Test
+ public void testOssFilesystemProperties() throws Exception {
+ Map<String, String> properties = new HashMap<>();
+ properties.put("type", "lance");
+ properties.put(LanceFileSystemMetastoreProperties.WAREHOUSE,
+ "oss://bucket/lance");
+
+ AbstractLanceProperties lanceProperties =
+ (AbstractLanceProperties)
MetastoreProperties.create(properties);
+
+ Assertions.assertInstanceOf(LanceFileSystemMetastoreProperties.class,
lanceProperties);
+ Assertions.assertEquals("oss://bucket/lance",
+ ((LanceFileSystemMetastoreProperties)
lanceProperties).getWarehouse());
+ }
+
@Test
public void testRestProperties() throws Exception {
Map<String, String> properties = new HashMap<>();
@@ -60,4 +75,35 @@ public class LancePropertiesTest {
Assertions.assertEquals("http://localhost:8080";,
restProperties.getRestUri());
Assertions.assertEquals("bearer", restProperties.getSecurityType());
}
+
+ @Test
+ public void testObjectStoreWarehouseMustNameABucket() {
+ for (String warehouse : new String[] {"oss:/lance", "s3:/lance"}) {
+ Map<String, String> properties = new HashMap<>();
+ properties.put("type", "lance");
+ properties.put(LanceFileSystemMetastoreProperties.WAREHOUSE,
warehouse);
+
+ IllegalArgumentException thrown = Assertions.assertThrows(
+ IllegalArgumentException.class, () ->
MetastoreProperties.create(properties));
+ Assertions.assertTrue(thrown.getMessage().contains("must name a
bucket"),
+ "unexpected message: " + thrown.getMessage());
+ }
+ }
+
+ /**
+ * OSSHdfsProperties selects on the endpoint, so a clean-looking warehouse
still routes there
+ * when the endpoint names OSS-HDFS. Checking only the warehouse authority
would miss it.
+ */
+ @Test
+ public void testOssHdfsEndpointIsRejectedEvenWithAPlainWarehouse() {
+ Map<String, String> properties = new HashMap<>();
+ properties.put("type", "lance");
+ properties.put(LanceFileSystemMetastoreProperties.WAREHOUSE,
"oss://bkt/lance");
+ properties.put("oss.endpoint", "cn-hangzhou.oss-dls.aliyuncs.com");
+
+ IllegalArgumentException thrown = Assertions.assertThrows(
+ IllegalArgumentException.class, () ->
MetastoreProperties.create(properties));
+ Assertions.assertTrue(thrown.getMessage().contains("OSS-HDFS is not
supported"),
+ "unexpected message: " + thrown.getMessage());
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
