szehon-ho commented on code in PR #17954:
URL: https://github.com/apache/iceberg/pull/17954#discussion_r4031351427
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -206,11 +212,55 @@ public Table createTable(
Identifier ident, StructType schema, Transform[] transforms, Map<String,
String> properties)
throws TableAlreadyExistsException {
Schema icebergSchema = SparkSchemaUtil.convert(schema);
+ return createTable(ident, icebergSchema, transforms, properties,
SortOrder.unsorted());
+ }
+
+ @Override
+ public Table createTableLike(Identifier ident, TableInfo tableInfo, Table
sourceTable)
+ throws TableAlreadyExistsException, NoSuchNamespaceException {
+ // Spark intentionally excludes the source table's properties from
tableInfo and leaves it to
+ // the connector to decide which to clone via sourceTable. Clone the
source Iceberg table's
+ // schema, properties and sort order, then let user-specified LIKE options
(in tableInfo) take
+ // precedence.
+ Schema icebergSchema;
+ Map<String, String> properties = Maps.newHashMap();
+ SortOrder sortOrder = SortOrder.unsorted();
+
+ if (sourceTable instanceof SparkTable) {
+ org.apache.iceberg.Table sourceIcebergTable = ((SparkTable)
sourceTable).table();
+ icebergSchema = sourceIcebergTable.schema();
Review Comment:
Please preserve the effective Iceberg schema represented by `SparkTable`,
rather than always using the current table schema. For a `tag_*`,
`snapshot_id_*`, or timestamp selector, `table().schema()` can differ from the
selected snapshot’s schema.
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -206,11 +212,55 @@ public Table createTable(
Identifier ident, StructType schema, Transform[] transforms, Map<String,
String> properties)
throws TableAlreadyExistsException {
Schema icebergSchema = SparkSchemaUtil.convert(schema);
+ return createTable(ident, icebergSchema, transforms, properties,
SortOrder.unsorted());
+ }
+
+ @Override
+ public Table createTableLike(Identifier ident, TableInfo tableInfo, Table
sourceTable)
+ throws TableAlreadyExistsException, NoSuchNamespaceException {
+ // Spark intentionally excludes the source table's properties from
tableInfo and leaves it to
+ // the connector to decide which to clone via sourceTable. Clone the
source Iceberg table's
+ // schema, properties and sort order, then let user-specified LIKE options
(in tableInfo) take
+ // precedence.
+ Schema icebergSchema;
+ Map<String, String> properties = Maps.newHashMap();
+ SortOrder sortOrder = SortOrder.unsorted();
+
+ if (sourceTable instanceof SparkTable) {
+ org.apache.iceberg.Table sourceIcebergTable = ((SparkTable)
sourceTable).table();
+ icebergSchema = sourceIcebergTable.schema();
+ properties.putAll(sourceIcebergTable.properties());
+ properties.remove(TableProperties.WRITE_METADATA_LOCATION);
+ properties.remove(TableProperties.WRITE_DATA_LOCATION);
+ properties.remove(TableProperties.OBJECT_STORE_PATH);
+ properties.remove(TableProperties.WRITE_FOLDER_STORAGE_LOCATION);
+ properties.put(
+ TableProperties.FORMAT_VERSION,
+ String.valueOf(TableUtil.formatVersion(sourceIcebergTable)));
+ sortOrder =
+ copySortOrder(sourceIcebergTable.schema(), icebergSchema,
sourceIcebergTable.sortOrder());
+ } else {
+ icebergSchema = SparkSchemaUtil.convert(tableInfo.schema());
+ }
+
+ properties.putAll(tableInfo.properties());
+
+ return createTable(ident, icebergSchema, tableInfo.partitions(),
properties, sortOrder);
Review Comment:
Please copy the source Iceberg partition spec directly instead of
round-tripping through Spark transforms. That conversion drops custom
partition-field names such as `bucket(16, id) AS shard` and rejects unknown
transforms.
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/SparkSessionCatalog.java:
##########
@@ -252,6 +253,19 @@ public Table createTable(
}
}
+ @Override
+ public Table createTableLike(Identifier ident, TableInfo tableInfo, Table
sourceTable)
+ throws TableAlreadyExistsException, NoSuchNamespaceException {
+ checkViewNotExists(ident);
+
+ String provider = tableInfo.properties().get("provider");
Review Comment:
Please fall back to the source table’s provider when `USING` is omitted.
Otherwise `provider` is null and `useIceberg` always routes a Parquet/Avro/ORC
source to Iceberg, even when the corresponding conversion option is disabled.
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -206,11 +212,55 @@ public Table createTable(
Identifier ident, StructType schema, Transform[] transforms, Map<String,
String> properties)
throws TableAlreadyExistsException {
Schema icebergSchema = SparkSchemaUtil.convert(schema);
+ return createTable(ident, icebergSchema, transforms, properties,
SortOrder.unsorted());
+ }
+
+ @Override
+ public Table createTableLike(Identifier ident, TableInfo tableInfo, Table
sourceTable)
+ throws TableAlreadyExistsException, NoSuchNamespaceException {
+ // Spark intentionally excludes the source table's properties from
tableInfo and leaves it to
+ // the connector to decide which to clone via sourceTable. Clone the
source Iceberg table's
+ // schema, properties and sort order, then let user-specified LIKE options
(in tableInfo) take
+ // precedence.
+ Schema icebergSchema;
+ Map<String, String> properties = Maps.newHashMap();
+ SortOrder sortOrder = SortOrder.unsorted();
+
+ if (sourceTable instanceof SparkTable) {
+ org.apache.iceberg.Table sourceIcebergTable = ((SparkTable)
sourceTable).table();
+ icebergSchema = sourceIcebergTable.schema();
+ properties.putAll(sourceIcebergTable.properties());
Review Comment:
Please rebuild or omit `schema.name-mapping.default` before copying source
properties. New-table creation assigns fresh field IDs, so an evolved source
schema can leave the target’s name mapping pointing at IDs that do not exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
