Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-12-29 Thread via GitHub
zzeekk commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2564779267 how can i reference parameters in an ArgoCD template? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-11-06 Thread via GitHub
github-actions[bot] commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2461060320 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-11-06 Thread via GitHub
github-actions[bot] closed issue #9990: Spark can not delete table metadata and data when drop table URL: https://github.com/apache/iceberg/issues/9990 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-10-22 Thread via GitHub
github-actions[bot] commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2430545215 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-04-11 Thread via GitHub
wForget commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2049244909 I also encountered the same problem when using SparkSessionCatalog to delete a non-iceberg hive table. @manuzhang Iceberg HiveCatalog allows deletion of non-iceberg tables, is

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-25 Thread via GitHub
wanghualei commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2019248320 > > Because it undermines table level integrity > > Iceberg manages table integrity. What can be improved is to offer options to delete directory when users know it's safe t

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-25 Thread via GitHub
manuzhang commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2019189367 > Because it undermines table level integrity Iceberg manages table integrity. What can be improved is to offer options to delete directory when users know it's safe to do so.

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-25 Thread via GitHub
wanghualei commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2019148264 > For example, I can create a table B with location under that of table A. I don't want to delete table B when dropping table A. If we reason like this, does HadoopCatalog a

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-25 Thread via GitHub
wanghualei commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2019145436 > For example, I can create a table B with location under that of table A. I don't want to delete table B when dropping table A. Generally speaking, this situation should no

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-25 Thread via GitHub
manuzhang commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2017943260 For example, I can create a table B with location under that of table A. I don't want to delete table B when dropping table A. -- This is an automated message from the Apache Git

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-25 Thread via GitHub
wanghualei commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2017939050 Is it related to object storage? There is no directory relationship. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-25 Thread via GitHub
wanghualei commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2017924722 @manuzhang > @tomfans If you mean empty table directories are left over, I can confirm that's behavior for `HiveCatalog`. It removes the table record from metastore, and

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-25 Thread via GitHub
manuzhang commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2017559451 You may create external auto-purge process if you are sure these directories are safe to delete. -- This is an automated message from the Apache Git Service. To respond to the me

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-25 Thread via GitHub
tomfans commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2017440800 > deleting yes, it's HiveCatalog, hadoopcatalog works fine. you mean this is normal operation for HiveCatalog to keep these data/metadata/table directories when drop table ?

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-25 Thread via GitHub
manuzhang commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2017435371 @tomfans If you mean empty table directories are left over, I can confirm that's behavior for `HiveCatalog`. It removes the table record from metastore, and deletes all referenced

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-25 Thread via GitHub
tomfans commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2017410644 > I meet this problem also. when user "drop table purge", it delete files in data and meta directory,but ,the table driectory still exits, in normal sence , the directory should be d

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-23 Thread via GitHub
wanghualei commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2016525287 I meet this problem also. when user "drop table purge", it delete files in data and meta directory,but ,the table driectory still exits, in normal sence , the directory should b

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-19 Thread via GitHub
tomfans commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2008661653 i just want to confirm while i use spark with iceberg catalog "org.apache.iceberg.spark.SparkCatalog" and HMS, if it's normal or not when it can't delete table directories after

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-19 Thread via GitHub
manuzhang commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2008655496 > alter table from external table to managed table This only works with Hive CLI. -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-19 Thread via GitHub
tomfans commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2008641881 and i have checked other comments on this kind issue , why can't delete table directories when drop table , it's because the table is a external table when created by spark, and givi

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-19 Thread via GitHub
tomfans commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2008639451 HMS, if i use HMS catalog , i can't delete table directories when i drop table, even i drop table with purge. the software version is: spark 3.3.2, hive 2.3.9, iceberg jar is:1.4.0.

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-19 Thread via GitHub
manuzhang commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2008591571 Which catalogs do you use in those two cases? Can you share the configs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [I] Spark can not delete table metadata and data when drop table [iceberg]

2024-03-18 Thread via GitHub
tomfans commented on issue #9990: URL: https://github.com/apache/iceberg/issues/9990#issuecomment-2003359267 more information: metadata managed by HMS, not HDFS. it works when metadata managed by HDFS -- This is an automated message from the Apache Git Service. To respond to the message