rahulsmahadev commented on code in PR #17954:
URL: https://github.com/apache/iceberg/pull/17954#discussion_r4225925529
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -206,11 +213,64 @@ public Table createTable(
Identifier ident, StructType schema, Transform[] transforms, Map<String,
String> properties)
throws TableAlreadyExistsException {
Schema icebergSchema = SparkSchemaUtil.convert(schema);
+ return createTable(
+ ident,
+ icebergSchema,
+ Spark3Util.toPartitionSpec(icebergSchema, transforms),
+ properties,
+ SortOrder.unsorted());
+ }
+
+ @Override
+ public Table createTableLike(Identifier ident, TableInfo tableInfo, Table
sourceTable)
+ throws TableAlreadyExistsException, NoSuchNamespaceException {
+ // Spark intentionally excludes the source table's properties from
tableInfo and leaves it to
+ // the connector to decide which to clone via sourceTable. Clone the
source Iceberg table's
+ // schema, partition spec, properties and sort order, then let
user-specified LIKE options (in
+ // tableInfo) take precedence.
+ Schema icebergSchema;
+ PartitionSpec spec;
+ Map<String, String> properties = Maps.newHashMap();
+ SortOrder sortOrder = SortOrder.unsorted();
+
+ if (sourceTable instanceof SparkTable sparkTable) {
+ org.apache.iceberg.Table sourceIcebergTable = sparkTable.table();
+ icebergSchema = sparkTable.icebergSchema();
+ spec = sourceIcebergTable.spec();
+ properties.putAll(sourceIcebergTable.properties());
+ properties.remove(TableProperties.WRITE_METADATA_LOCATION);
+ properties.remove(TableProperties.WRITE_DATA_LOCATION);
+ properties.remove(TableProperties.OBJECT_STORE_PATH);
+ properties.remove(TableProperties.WRITE_FOLDER_STORAGE_LOCATION);
+ properties.remove(TableProperties.DEFAULT_NAME_MAPPING);
+ properties.put(
+ TableProperties.FORMAT_VERSION,
+ String.valueOf(TableUtil.formatVersion(sourceIcebergTable)));
+ sortOrder =
+ copySortOrder(sourceIcebergTable.schema(), icebergSchema,
sourceIcebergTable.sortOrder());
+ } else {
+ icebergSchema = SparkSchemaUtil.convert(tableInfo.schema());
Review Comment:
Following your later comment, non-Iceberg sources now go through
SparkSchemaUtil.convertWithDefaults(tableInfo), and the custom default handling
is gone.
One gap to flag: for a non-Iceberg source, Spark passes the default as SQL
text with no typed expression, and convertWithDefaults rejects that.
So cloning a parquet source with DEFAULT 42 fails with "Unsupported default
value expression: 42". The default isn't silently lost, but it isn't preserved
either, and there's no test for this path yet. Supporting it looks like it
needs either the SQL parsing we just removed or Spark passing a typed
expression.
Would you rather keep the explicit failure and have me add a test for it, or
handle it in a follow-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
