hidataplus commented on PR #13497:
URL: https://github.com/apache/iceberg/pull/13497#issuecomment-3111656516
> @hidataplus: Could you please describe what is the difference when
Spark3.5 drops a table? What is the parametrization of the
`dropTable(TableIdentifier, boolean)` method when it is called from Spark? Is
the difference caused by the `purge` flag is set to `false` when Spark is used?
>
> A few general observation:
>
> * We should not introduce behavioral change. Even if we provide a
configuration to revert back to the previous behavior. If we decide to
introduce a flag like `delete-table-rootdir`, then it should default to `false`
to preserve the pervious behavior.
> * Don't introduce so many `INFO` level logs. A single `INFO` on a delete,
and few `DEBUG` level logs might be ok, but we don't want to introduce this
many new log lines.
@pvary
when create a iceberg table using hive catalog in Hive4, if you drop table
case 1: in hive4 beeline, it will drop table metadata in hivemetastore, it
will also delete the table store path (the table root hdfs directory);
case 2: in spark 3.5 , it will drop table metadata in hivemetastore, it
will delete the table refer files ( recored in icebert meta.json), but it will
not delete the directory (the table root path and under the path), it will
make confusing.
we found in the iceberg code, it's logic when drop table metastore , it call
method :
client.dropTable(
database,
identifier.name(),
false /* do not delete data */,
deleteData,
so it will not drop table data through metastore, and it will delte
datafiles using iceberg meta info. it diffent with hive4.
and
1. delete-table-rootdir set to false default;
2. remove INFO level logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
