This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e83b06cb7b7 [fix](iceberg) Let MODIFY COLUMN restate Iceberg BINARY
and FIXED columns as VARBINARY (#69285)
e83b06cb7b7 is described below
commit e83b06cb7b761ffa7ebed01412df90f6652306bb
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Oct 21 08:20:39 2026 +0100
[fix](iceberg) Let MODIFY COLUMN restate Iceberg BINARY and FIXED columns
as VARBINARY (#69285)
### What problem does this PR solve?
Issue Number: None
Related PR: #66032, #68786
Problem Summary:
**In short.** This ports to master the rule branch-4.1 got in #66032
(`IcebergMetadataOps.resolvePrimitiveTypeForModify`): when `ALTER TABLE
... MODIFY COLUMN` restates the type an Iceberg column is read as, the
Iceberg type stays as it is. On master, every MODIFY COLUMN of an
Iceberg BINARY or FIXED column, or of such a STRUCT field, ARRAY element
or MAP value, fails with `Unsupported type for Iceberg: VARBINARY`, so
their nullability, comment and position cannot be changed at all. After
this PR they can.
**Background.** Since #68786, Doris always reads an Iceberg BINARY
column as `VARBINARY` and a FIXED(n) column as `VARBINARY(n)`. MODIFY
COLUMN carries the whole column definition, so a statement that only
means to change the nullability, comment or position of such a column
restates that type, for example `MODIFY COLUMN payload VARBINARY NULL
FIRST`. The nested form (`MODIFY COLUMN s.a ...`, `arr.element`,
`m.value`) works the same way. The Iceberg connector turns the requested
type into an Iceberg type before it looks at the table
(`IcebergSchemaBuilder.buildColumnType`), and then diffs it against the
current schema. `VARBINARY` has no Iceberg type there: CREATE TABLE and
ADD COLUMN reject it on purpose, as branch-4.x does, because a Doris
`VARBINARY(n)` holds up to n bytes and an Iceberg FIXED(n) exactly n.
**The problem, and what it cost.** The type build fails before the
connector ever looks at the current column, so restating the type fails
too:
| Statement (current Iceberg type) | master | this PR |
|---|---|---|
| `MODIFY COLUMN payload VARBINARY NULL FIRST` (BINARY) | `Unsupported
type for Iceberg: VARBINARY` | succeeds, stays BINARY |
| `MODIFY COLUMN digest VARBINARY(16) NULL COMMENT 'digest'` (required
FIXED(16)) | same error | succeeds, stays FIXED(16) |
| `MODIFY COLUMN s.a VARBINARY COMMENT 'bytes'` (STRUCT field BINARY) |
same error | succeeds, stays BINARY |
| `MODIFY COLUMN arr.element VARBINARY(4) NULL` (required FIXED(4)
element) | same error | succeeds, stays FIXED(4) |
| `MODIFY COLUMN m.value VARBINARY NULL` (required BINARY value) | same
error | succeeds, stays BINARY |
| `MODIFY COLUMN payload VARBINARY(16)` (BINARY: not how it is read) |
`Unsupported type for Iceberg: VARBINARY` | unchanged |
So a column that users read and write as VARBINARY could not be moved,
commented, or made nullable from Doris; only a comment-only `MODIFY
COLUMN ... COMMENT '...'` worked, because it carries no type. On
branch-4.1 and branch-4.2,
`IcebergMetadataOps.resolvePrimitiveTypeForModify` keeps the current
Iceberg type in the first five cases and rejects the last one (`Type
varbinary(16) is not supported for Iceberg column payload`).
**How this PR fixes it.** When the requested type does not build and it
is a `VARBINARY`, `IcebergConnectorMetadata` looks up the current type
of the target field (the same best-effort lookup it already did to
rewrite the nested error message) and keeps it if that type is read as
exactly the requested `VARBINARY`, length included
(`IcebergTypeMapping.isReadAs`). The schema update then sees an
unchanged type and applies only the nullability, comment and position
the statement asks for. This is the same check the Paimon connector
already makes in `requestedColumnType`. A `VARBINARY` that restates
nothing (another length, a column or field of another type) still fails
with `Unsupported type for Iceberg: VARBINARY`, as ADD COLUMN does; any
other scalar that does not build still fails without loading the table,
and a complex type still gets the nested error message as before. A
STRUCT, ARRAY or MAP type that contains `VARBINARY` is rejected by
Doris's DDL type check (`STRUCT unsupported sub-type: varbinary(...)`)
before it reaches the connector, on branch-4.x as well, so it is
unchanged.
**Results.**
- **Regression test**: new
`external_table_p0/iceberg/test_iceberg_modify_binary_column`. The table
is created through the REST catalog, since neither Doris nor Spark SQL
can declare a FIXED column. The suite runs the five statements above,
checks with `DESC` (comments shown) that the types are unchanged while
the position and comments changed, checks that the required column,
ARRAY element and MAP value reject NULL before and accept it after, and
compares the rows with what Spark reads. It fails at its first MODIFY
with the connector before this PR and passes with it. Seven related
suites pass as well: `test_iceberg_schema_change_complex_types`,
`test_iceberg_nested_schema_evolution_ddl`,
`test_iceberg_nested_schema_evolution_spark_doris_interop`,
`iceberg_schema_change_ddl`, `test_iceberg_varbinary`,
`test_iceberg_uuid_binary_compatibility` and
`test_iceberg_struct_schema_evolution`.
- **FE UT**: new `IcebergConnectorMetadataBinaryModifyTest` drives
`IcebergConnectorMetadata` into an in-memory Iceberg catalog. Its four
restating cases fail before this PR with `Unsupported type for Iceberg:
VARBINARY`; the fifth checks that a VARBINARY that restates nothing
still fails and leaves the schema unchanged. Turning off the length
comparison, the primitive check, the null check, or reading the current
type at the top-level column instead of the nested path each turns a
case red. All 1615 tests of fe-connector-iceberg pass.
Co-authored-by: Gabriel <[email protected]>
---
.../iceberg/IcebergConnectorMetadata.java | 62 +++++--
.../iceberg/IcebergNestedColumnEvolution.java | 5 +-
.../connector/iceberg/IcebergTypeMapping.java | 11 ++
.../IcebergConnectorMetadataBinaryModifyTest.java | 179 +++++++++++++++++++++
.../iceberg/test_iceberg_modify_binary_column.out | 21 +++
.../test_iceberg_modify_binary_column.groovy | 138 ++++++++++++++++
6 files changed, 398 insertions(+), 18 deletions(-)
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
index 43901064454..3864846d2b3 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
@@ -1392,11 +1392,11 @@ public class IcebergConnectorMetadata implements
ConnectorMetadata {
try {
icebergType =
IcebergSchemaBuilder.buildColumnType(column.getType());
} catch (DorisConnectorException buildError) {
- // A nested narrowing to an iceberg-unrepresentable type (e.g.
ARRAY<INT> -> ARRAY<SMALLINT>) throws a
- // generic "Unsupported type for Iceberg: SMALLINT" here. Restore
the legacy parity message ("Cannot
- // change int to smallint in nested types") by validating the
requested nested type against the
- // CURRENT type — legacy validated in Doris type space, where the
narrow target still exists.
- throw upgradeNestedModifyError(iceHandle,
ConnectorColumnPath.of(column.getName()), column, buildError);
+ // VARBINARY, how iceberg BINARY / FIXED is read, has no iceberg
type of its own, and a nested narrowing
+ // to an iceberg-unrepresentable type (e.g. ARRAY<INT> ->
ARRAY<SMALLINT>) throws a generic
+ // "Unsupported type for Iceberg: SMALLINT" here. Both are
resolved against the CURRENT type.
+ icebergType = keepRestatedBinaryType(iceHandle,
ConnectorColumnPath.of(column.getName()), column,
+ buildError);
}
// Carry the neutral source type so a complex-type diff can read each
STRUCT field's commentSpecified.
IcebergColumnChange change = new IcebergColumnChange(column.getName(),
icebergType,
@@ -1415,6 +1415,25 @@ public class IcebergConnectorMetadata implements
ConnectorMetadata {
}
}
+ /**
+ * Keeps the CURRENT iceberg type of a {@code MODIFY COLUMN} target that
the requested VARBINARY restates. Doris
+ * reads iceberg BINARY and FIXED(n) as VARBINARY and VARBINARY(n), which
have no iceberg type of their own, so
+ * restating how such a column or field is read is the only way to change
its nullability, comment or
+ * position (branch-4.x {@code
IcebergMetadataOps.resolvePrimitiveTypeForModify} keeps the iceberg type the
same
+ * way). Any other requested type that did not build gets {@link
#upgradeNestedModifyError}.
+ */
+ private Type keepRestatedBinaryType(IcebergTableHandle handle,
ConnectorColumnPath path,
+ ConnectorColumn column, DorisConnectorException buildError) {
+ if
("VARBINARY".equals(column.getType().getTypeName().toUpperCase(Locale.ROOT))) {
+ Types.NestedField current = loadModifyTarget(handle, path);
+ if (current != null && current.type().isPrimitiveType()
+ &&
IcebergTypeMapping.isReadAs(current.type().asPrimitiveType(),
column.getType())) {
+ return current.type();
+ }
+ }
+ throw upgradeNestedModifyError(handle, path, column, buildError);
+ }
+
/**
* Upgrades the generic "Unsupported type for Iceberg" error from a failed
complex-type build into the legacy
* "Cannot change <old> to <new> in nested types" message, by
walking the requested nested type
@@ -1426,21 +1445,31 @@ public class IcebergConnectorMetadata implements
ConnectorMetadata {
if (!isComplexType(column.getType())) {
return buildError;
}
- try {
- Types.NestedField current = executeAuthenticated(() ->
- catalogOps.withTable(handle.getDbName(),
handle.getTableName(),
- table ->
IcebergNestedColumnEvolution.findFieldForErrorUpgrade(table.schema(), path)));
- if (current != null && !current.type().isPrimitiveType()) {
+ Types.NestedField current = loadModifyTarget(handle, path);
+ if (current != null && !current.type().isPrimitiveType()) {
+ try {
IcebergComplexTypeDiff.validateNestedModifyRepresentable(current.type(),
column.getType());
+ } catch (DorisConnectorException parityError) {
+ return parityError;
}
- } catch (DorisConnectorException parityError) {
- return parityError;
- } catch (Exception ignored) {
- // load failed / column missing -> keep the original build error
}
return buildError;
}
+ /**
+ * The current field at {@code path}, read only to resolve a requested
type that did not build: null when the
+ * table does not load or has no such field, which keeps the original
build error.
+ */
+ private Types.NestedField loadModifyTarget(IcebergTableHandle handle,
ConnectorColumnPath path) {
+ try {
+ return executeAuthenticated(() ->
catalogOps.withTable(handle.getDbName(), handle.getTableName(),
+ table ->
IcebergNestedColumnEvolution.findModifyTarget(table.schema(), path)));
+ } catch (Exception ignored) {
+ // load failed -> keep the original build error
+ return null;
+ }
+ }
+
/** Reorders columns, mirroring legacy {@code
IcebergMetadataOps.reorderColumns}. */
@Override
public void reorderColumns(ConnectorSession session, ConnectorTableHandle
handle, List<String> newOrder) {
@@ -1580,8 +1609,9 @@ public class IcebergConnectorMetadata implements
ConnectorMetadata {
try {
icebergType =
IcebergSchemaBuilder.buildNestedFieldType(column.getType());
} catch (DorisConnectorException buildError) {
- // Preserve the complete target identity so error parity cannot
bind a same-named top-level field.
- throw upgradeNestedModifyError(iceHandle, path, column,
buildError);
+ // Preserve the complete target identity so the current type
cannot be read from a same-named
+ // top-level field.
+ icebergType = keepRestatedBinaryType(iceHandle, path, column,
buildError);
}
// Carry the neutral source type so the nested complex-type diff can
read each STRUCT field's
// commentSpecified (an omitted COMMENT on a sub-field must keep its
current doc, not clear it).
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolution.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolution.java
index 7bc5a0e52b8..cd97c21b4a8 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolution.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolution.java
@@ -496,11 +496,12 @@ public final class IcebergNestedColumnEvolution {
return schema.asStruct().caseInsensitiveField(columnName);
}
- static NestedField findFieldForErrorUpgrade(Schema schema,
ConnectorColumnPath columnPath) {
+ static NestedField findModifyTarget(Schema schema, ConnectorColumnPath
columnPath) {
try {
return resolveColumnPath(schema, columnPath, "modify").getField();
} catch (DorisConnectorException ignored) {
- // Error-message upgrading is best-effort and must not replace the
original type-build failure.
+ // Building a modify against the current type is best-effort and
must not replace the original
+ // type-build failure.
return null;
}
}
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java
index 7192ae2671d..39d6bb63f11 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java
@@ -171,6 +171,17 @@ public final class IcebergTypeMapping {
}
}
+ /**
+ * Whether the primitive {@code icebergType} is read as {@code type}, so a
{@code MODIFY COLUMN} stating
+ * {@code type} restates it rather than changing it. The read mapping
cannot always be inverted by
+ * {@link #toIcebergPrimitive}: iceberg BINARY and FIXED(n) are read as
VARBINARY and VARBINARY(n), which
+ * have no iceberg type of their own.
+ */
+ static boolean isReadAs(Type.PrimitiveType icebergType, ConnectorType
type) {
+ // Neither legacy mapping option changes how a primitive type is read.
+ return type.equals(fromPrimitive(icebergType, true, true));
+ }
+
/**
* Maps a SCALAR {@link ConnectorType} (a Doris column type carried across
the SPI by name) to an
* Iceberg {@link Type} for CREATE TABLE. The inverse of {@link
#fromPrimitive}, this is a string-driven
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataBinaryModifyTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataBinaryModifyTest.java
new file mode 100644
index 00000000000..64b084df03a
--- /dev/null
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataBinaryModifyTest.java
@@ -0,0 +1,179 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.iceberg;
+
+import
org.apache.doris.connector.iceberg.IcebergCatalogOps.CatalogBackedIcebergCatalogOps;
+import org.apache.doris.connector.spi.ConnectorColumn;
+import org.apache.doris.connector.spi.ConnectorType;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.ddl.ConnectorColumnPath;
+import org.apache.doris.connector.spi.ddl.ConnectorColumnPosition;
+
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * {@code MODIFY COLUMN} of iceberg BINARY / FIXED columns and nested fields,
driven through
+ * {@link IcebergConnectorMetadata} into a REAL {@link InMemoryCatalog}. Doris
reads BINARY as {@code VARBINARY} and
+ * FIXED(n) as {@code VARBINARY(n)}, and iceberg has no type to build for
VARBINARY, so a modify that restates how
+ * such a column or field is read (to change its nullability, comment or
position) must keep the current iceberg
+ * type rather than fail with "Unsupported type for Iceberg: VARBINARY". A
VARBINARY that restates nothing still
+ * fails.
+ */
+public class IcebergConnectorMetadataBinaryModifyTest {
+
+ private static final IcebergTableHandle HANDLE = new
IcebergTableHandle("db1", "t1");
+ private static final String UNSUPPORTED = "Unsupported type for Iceberg:
VARBINARY";
+
+ private InMemoryCatalog catalog;
+ private CatalogBackedIcebergCatalogOps ops;
+ private IcebergConnectorMetadata metadata;
+
+ @BeforeEach
+ public void setUp() {
+ catalog = new InMemoryCatalog();
+ catalog.initialize("test", Collections.emptyMap());
+ ops = new CatalogBackedIcebergCatalogOps(catalog);
+ ops.createDatabase("db1", Collections.emptyMap());
+ Schema schema = new Schema(
+ Types.NestedField.optional(1, "id", Types.LongType.get()),
+ Types.NestedField.optional(2, "payload",
Types.BinaryType.get()),
+ Types.NestedField.required(3, "digest",
Types.FixedType.ofLength(16), "sha"),
+ Types.NestedField.optional(4, "s", Types.StructType.of(
+ Types.NestedField.optional(5, "a",
Types.BinaryType.get(), "raw"),
+ Types.NestedField.optional(6, "b",
Types.IntegerType.get()))),
+ Types.NestedField.optional(7, "arr",
Types.ListType.ofRequired(8, Types.FixedType.ofLength(4))),
+ Types.NestedField.optional(9, "m",
Types.MapType.ofRequired(10, 11,
+ Types.StringType.get(), Types.BinaryType.get())));
+ ops.createTable("db1", "t1", schema, PartitionSpec.unpartitioned(),
null,
+
IcebergSchemaBuilder.buildTableProperties(Collections.emptyMap()));
+ Map<String, String> props = new HashMap<>();
+ props.put(IcebergCatalogProperties.ICEBERG_CATALOG_TYPE,
IcebergCatalogProperties.TYPE_REST);
+ metadata = new IcebergConnectorMetadata(ops,
IcebergCatalogProperties.of(props),
+ new RecordingConnectorContext());
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ catalog.close();
+ }
+
+ private Schema reload() {
+ return ops.loadTable("db1", "t1").schema();
+ }
+
+ /** The neutral type fe-core sends for a Doris {@code VARBINARY}, which is
how iceberg BINARY is read. */
+ private static ConnectorType varbinary() {
+ return ConnectorType.of("VARBINARY");
+ }
+
+ /** The neutral type fe-core sends for a Doris {@code VARBINARY(n)}, which
is how iceberg FIXED(n) is read. */
+ private static ConnectorType varbinary(int length) {
+ return ConnectorType.of("VARBINARY", length, 0);
+ }
+
+ /** A nullable column with its nullability stated and its comment omitted.
*/
+ private static ConnectorColumn column(String name, ConnectorType type) {
+ return new ConnectorColumn(name, type, "", true, null,
false).withSpecified(true, false);
+ }
+
+ private static ConnectorColumnPath path(String... parts) {
+ return ConnectorColumnPath.of(Arrays.asList(parts));
+ }
+
+ @Test
+ public void testRestatingBinaryColumnKeepsBinaryAndMovesIt() {
+ // MODIFY COLUMN payload VARBINARY NULL FIRST
+ metadata.modifyColumn(null, HANDLE, column("payload", varbinary()),
ConnectorColumnPosition.FIRST);
+
+ Schema schema = reload();
+ Assertions.assertEquals(Types.BinaryType.get(),
schema.findField("payload").type());
+ Assertions.assertEquals("payload", schema.columns().get(0).name());
+ }
+
+ @Test
+ public void testRestatingFixedColumnKeepsFixedAndWidensNullability() {
+ // MODIFY COLUMN digest VARBINARY(16) NULL COMMENT 'digest'
+ ConnectorColumn digest = new ConnectorColumn("digest", varbinary(16),
"digest", true, null, false)
+ .withSpecified(true, true);
+ metadata.modifyColumn(null, HANDLE, digest, null);
+
+ Types.NestedField field = reload().findField("digest");
+ Assertions.assertEquals(Types.FixedType.ofLength(16), field.type());
+ Assertions.assertTrue(field.isOptional());
+ Assertions.assertEquals("digest", field.doc());
+ }
+
+ @Test
+ public void testRestatingBinaryStructFieldChangesItsComment() {
+ // MODIFY COLUMN s.a VARBINARY COMMENT 'bytes'
+ ConnectorColumn a = new ConnectorColumn("a", varbinary(), "bytes",
true, null, false)
+ .withSpecified(false, true);
+ metadata.modifyNestedColumn(null, HANDLE, path("s", "a"), a, null);
+
+ Types.NestedField field = reload().findField("s.a");
+ Assertions.assertEquals(Types.BinaryType.get(), field.type());
+ Assertions.assertEquals("bytes", field.doc());
+ }
+
+ @Test
+ public void testRestatingBinaryElementAndValueWidensThem() {
+ // MODIFY COLUMN arr.element VARBINARY(4) NULL; MODIFY COLUMN m.value
VARBINARY NULL
+ metadata.modifyNestedColumn(null, HANDLE, path("arr", "element"),
column("element", varbinary(4)), null);
+ metadata.modifyNestedColumn(null, HANDLE, path("m", "value"),
column("value", varbinary()), null);
+
+ Schema schema = reload();
+ Types.ListType arr = schema.findField("arr").type().asListType();
+ Assertions.assertEquals(Types.FixedType.ofLength(4),
arr.elementType());
+ Assertions.assertTrue(arr.isElementOptional());
+ Types.MapType m = schema.findField("m").type().asMapType();
+ Assertions.assertEquals(Types.BinaryType.get(), m.valueType());
+ Assertions.assertTrue(m.isValueOptional());
+ }
+
+ @Test
+ public void testVarbinaryThatRestatesNothingStillFails() {
+ Schema before = reload();
+ // A VARBINARY is not how the current column or field is read: a
length other than FIXED's, a length for an
+ // unbounded BINARY, a column or field of another type, or no such
column.
+ assertFails(() -> metadata.modifyColumn(null, HANDLE,
column("payload", varbinary(16)), null));
+ assertFails(() -> metadata.modifyColumn(null, HANDLE, column("digest",
varbinary(8)), null));
+ assertFails(() -> metadata.modifyColumn(null, HANDLE, column("id",
varbinary()), null));
+ assertFails(() -> metadata.modifyColumn(null, HANDLE, column("s",
varbinary()), null));
+ assertFails(() -> metadata.modifyColumn(null, HANDLE,
column("missing", varbinary()), null));
+ assertFails(() -> metadata.modifyNestedColumn(null, HANDLE, path("s",
"a"), column("a", varbinary(3)), null));
+ assertFails(() -> metadata.modifyNestedColumn(null, HANDLE, path("s",
"b"), column("b", varbinary()), null));
+ Assertions.assertTrue(before.sameSchema(reload()), "a failed modify
must not change the schema");
+ }
+
+ private static void assertFails(org.junit.jupiter.api.function.Executable
modify) {
+ DorisConnectorException ex =
Assertions.assertThrows(DorisConnectorException.class, modify);
+ Assertions.assertEquals(UNSUPPORTED, ex.getMessage());
+ }
+}
diff --git
a/regression-test/data/external_table_p0/iceberg/test_iceberg_modify_binary_column.out
b/regression-test/data/external_table_p0/iceberg/test_iceberg_modify_binary_column.out
new file mode 100644
index 00000000000..15b3bacfe57
--- /dev/null
+++
b/regression-test/data/external_table_p0/iceberg/test_iceberg_modify_binary_column.out
@@ -0,0 +1,21 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !desc_before --
+id int Yes true \N
+payload varbinary(2147483647) Yes true \N
+digest varbinary(16) Yes true \N sha
+s struct<a:varbinary(2147483647) comment "raw",b:int> Yes true
\N
+arr array<varbinary(4)> Yes true \N
+m map<text,varbinary(2147483647)> Yes true \N
+
+-- !desc_after --
+payload varbinary(2147483647) Yes true \N
+id int Yes true \N
+digest varbinary(16) Yes true \N digest
+s struct<a:varbinary(2147483647) comment "bytes",b:int> Yes true
\N
+arr array<varbinary(4)> Yes true \N
+m map<text,varbinary(2147483647)> Yes true \N
+
+-- !rows --
+1 0102 00112233445566778899AABBCCDDEEFF AB 1
01020304 02
+2 FF \N \N 2 \N \N
+
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_modify_binary_column.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_modify_binary_column.groovy
new file mode 100644
index 00000000000..e9487f4c04d
--- /dev/null
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_modify_binary_column.groovy
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import groovy.json.JsonOutput
+
+// Doris reads an Iceberg BINARY column as VARBINARY and a FIXED(n) column as
VARBINARY(n). MODIFY COLUMN carries
+// the whole column definition, so changing the nullability, comment or
position of such a column, or of such a
+// STRUCT field, ARRAY element or MAP value, restates that VARBINARY type, and
the Iceberg type must stay as it is.
+suite("test_iceberg_modify_binary_column",
"p0,external,iceberg,external_docker,external_docker_iceberg") {
+ String enabled = context.config.otherConfigs.get("enableIcebergTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable iceberg test.")
+ return
+ }
+
+ String catalogName = "test_iceberg_modify_binary_column"
+ String dbName = "iceberg_modify_binary_column_db"
+ String tableName = "binary_columns"
+ String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port")
+ String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+
+ sql """drop catalog if exists ${catalogName}"""
+ sql """
+ CREATE CATALOG ${catalogName} PROPERTIES (
+ 'type'='iceberg',
+ 'iceberg.catalog.type'='rest',
+ 'uri' = 'http://${externalEnvIp}:${restPort}',
+ "s3.access_key" = "admin",
+ "s3.secret_key" = "password",
+ "s3.endpoint" = "http://${externalEnvIp}:${minioPort}";,
+ "s3.region" = "us-east-1",
+ "meta.cache.iceberg.table.ttl-second" = "0"
+ );
+ """
+ sql """switch ${catalogName}"""
+ sql """drop database if exists ${dbName} force"""
+ sql """create database ${dbName}"""
+ sql """use ${dbName}"""
+
+ // Neither Doris (no Iceberg type for VARBINARY) nor Spark SQL (no FIXED
type) can declare these columns, so
+ // the table is created through the REST catalog.
+ def request = [name: tableName, schema: [type: "struct", "schema-id": 0,
fields: [
+ [id: 1, name: "id", required: false, type: "int"],
+ [id: 2, name: "payload", required: false, type: "binary"],
+ [id: 3, name: "digest", required: true, type: "fixed[16]", doc:
"sha"],
+ [id: 4, name: "s", required: false, type: [type: "struct", fields:
[
+ [id: 5, name: "a", required: false, type: "binary", doc:
"raw"],
+ [id: 6, name: "b", required: false, type: "int"]]]],
+ [id: 7, name: "arr", required: false,
+ type: [type: "list", "element-id": 8, element: "fixed[4]",
"element-required": true]],
+ [id: 9, name: "m", required: false, type: [type: "map", "key-id":
10, key: "string",
+ "value-id": 11, value: "binary", "value-required":
true]]]],
+ "partition-spec": ["spec-id": 0, fields: []],
+ properties: ["format-version": "2"]]
+ def connection = new
URL("http://${externalEnvIp}:${restPort}/v1/namespaces/${dbName}/tables";).openConnection()
+ try {
+ connection.setConnectTimeout(10000)
+ connection.setReadTimeout(60000)
+ connection.setRequestMethod("POST")
+ connection.setRequestProperty("Content-Type", "application/json")
+ connection.setDoOutput(true)
+ connection.getOutputStream().withCloseable {
it.write(JsonOutput.toJson(request).getBytes("UTF-8")) }
+ assertEquals(200, connection.getResponseCode())
+ } finally {
+ connection.disconnect()
+ }
+
+ sql """INSERT INTO ${tableName} VALUES (1, X'0102',
X'00112233445566778899AABBCCDDEEFF',
+ named_struct('a', X'AB', 'b', 1), [X'01020304'], map('k',
X'02'))"""
+ // DESC shows column and STRUCT field comments too.
+ sql """set show_column_comment_in_describe = true"""
+ qt_desc_before """DESC ${tableName}"""
+ // The required column, ARRAY element and MAP value reject NULL.
+ test {
+ sql """INSERT INTO ${tableName} VALUES (2, X'03', NULL, NULL, NULL,
NULL)"""
+ exception "Column 'digest' is declared non-nullable"
+ }
+ test {
+ sql """INSERT INTO ${tableName} VALUES (2, X'03',
X'00112233445566778899AABBCCDDEEFF', NULL, [NULL], NULL)"""
+ exception "Column 'element' is declared non-nullable"
+ }
+ test {
+ sql """INSERT INTO ${tableName} VALUES (2, X'03',
X'00112233445566778899AABBCCDDEEFF', NULL, NULL,
+ map('k', NULL))"""
+ exception "Column 'value' is declared non-nullable"
+ }
+
+ // Each statement restates the type DESC shows, to change the position,
nullability or comment.
+ sql """ALTER TABLE ${tableName} MODIFY COLUMN payload VARBINARY NULL
FIRST"""
+ sql """ALTER TABLE ${tableName} MODIFY COLUMN digest VARBINARY(16) NULL
COMMENT 'digest'"""
+ sql """ALTER TABLE ${tableName} MODIFY COLUMN s.a VARBINARY COMMENT
'bytes'"""
+ sql """ALTER TABLE ${tableName} MODIFY COLUMN arr.element VARBINARY(4)
NULL"""
+ sql """ALTER TABLE ${tableName} MODIFY COLUMN m.value VARBINARY NULL"""
+ // A VARBINARY that is not how the column or field is read has no Iceberg
type.
+ test {
+ sql """ALTER TABLE ${tableName} MODIFY COLUMN payload VARBINARY(16)"""
+ exception "Unsupported type for Iceberg: VARBINARY"
+ }
+ test {
+ sql """ALTER TABLE ${tableName} MODIFY COLUMN s.a VARBINARY(3)"""
+ exception "Unsupported type for Iceberg: VARBINARY"
+ }
+ // BINARY, FIXED(16) and FIXED(4) are still read the same way; payload
moved first and the comments changed.
+ qt_desc_after """DESC ${tableName}"""
+
+ // The column, ARRAY element and MAP value accept NULL now.
+ sql """INSERT INTO ${tableName} VALUES (X'FF', 2, NULL, named_struct('a',
NULL, 'b', 2), [NULL],
+ map('k', NULL))"""
+ String dorisQuery = """
+ SELECT id, hex(payload), hex(digest), hex(struct_element(s, 'a')),
struct_element(s, 'b'), hex(arr[1]),
+ hex(m['k'])
+ FROM ${tableName}
+ ORDER BY id
+ """
+ qt_rows dorisQuery
+ // Spark reads the evolved table back the same way.
+ def sparkRows = spark_iceberg """
+ SELECT id, hex(payload), hex(digest), hex(s.a), s.b, hex(arr[0]),
hex(m['k'])
+ FROM demo.${dbName}.${tableName}
+ ORDER BY id
+ """
+ assertSparkDorisResultEquals(sparkRows, sql(dorisQuery))
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
