myloginid opened a new issue, #4268:
URL: https://github.com/apache/amoro/issues/4268
### Summary
On secured Hadoop clusters, AMS and optimizers typically run as a
**dedicated service principal** (e.g. `amoro`). Iceberg/Hive warehouse paths
are often owned by the **table creator** (e.g. `lily`). Optimizing **tasks**
can succeed while the **AMS commit** fails with HDFS `AccessControlException`
on `data/` or `metadata/` because all catalog file IO runs as the service user,
not the table owner.
Configuring `hadoop.proxyuser.amoro.*` on the cluster is **necessary but not
sufficient**: nothing in the AMS optimizing commit path uses Hadoop proxy
impersonation today.
### Environment (example)
- Amoro 0.9.x (AMS + external optimizer)
- Kerberos, Hive Metastore + Iceberg tables on HDFS
- Ranger HDFS/Hive policies for the service user
- Tables under e.g. `/warehouse/tablespace/external/hive/<db>/<table>/` with
owner = end user, mode `drwxr-xr-x`
### Observed behavior
1. Optimizer finishes MINOR (or similar) work; UI shows tasks **1/1
success**.
2. AMS commit fails; optimizing process status **FAILED**.
3. AMS logs show `OptimizingCommitException` / Iceberg commit failure, root
cause similar to:
`Permission denied: user=amoro, access=WRITE,
inode=".../mor_events/metadata":lily:hive:drwxr-xr-x`
Ranger grants for `amoro` on `/*` and cluster
`hadoop.proxyuser.amoro.{hosts,groups,users}=*` did **not** change this,
because commits still execute as `amoro`.
Workarounds that work operationally but do not scale well:
- Per-table or per-DB HDFS ACLs for the service user on `data/` and
`metadata/`
- Warehouse-wide default ACLs
- Treating the service user as a broad HDFS/Ranger “power user”
These are painful in multi-tenant warehouses where table ownership follows
the creating user (as with Hive).
### Expected behavior
When a catalog property (or table property) enables **owner impersonation**,
AMS and optimizers should perform HDFS reads/writes for optimizing commits (and
related maintenance) **as the table owner** (from HMS/Iceberg metadata), using
the catalog’s Kerberos principal as the **proxy user**, consistent with
Hive/Impala-style deployments.
Cluster admins would configure:
```xml
<property><name>hadoop.proxyuser.amoro.hosts</name><value>*</value></property>
<property><name>hadoop.proxyuser.amoro.groups</name><value>*</value></property>
<property><name>hadoop.proxyuser.amoro.users</name><value>*</value></property>
```
(Exact host/group scope should follow cluster security policy.)
### Current implementation notes
- `TableMetaStore` already supports proxy impersonation via
`doAsImpersonating(String proxyUser, Callable<T>)`, which uses
`UserGroupInformation.createProxyUser(proxyUser, catalogUgi)`.
- `AuthenticatedHadoopFileIO` wraps HDFS access with
`tableMetaStore.doAs(...)` only — **not** `doAsImpersonating`.
- Trino integration is the main user-facing precedent:
`mixed-format.hdfs.impersonation.enabled` + `MixedFormatHdfsAuthentication`
calling `doAsImpersonating(identity.getUser(), ...)`. There is no equivalent
for **AMS optimizing commit** on Hive/Iceberg catalogs.
- Optimizing commit classes (e.g. `UnKeyedTableCommit`, `KeyedTableCommit`)
use table `FileIO` / `AuthenticatedFileIO.doAs()` under the catalog login user.
### Proposal
1. **Catalog-level config** (name bikeshedding welcome), e.g.
`hdfs.impersonation.enabled=true`
and optionally
`hdfs.impersonation.user-source=table-owner` (HMS owner / Iceberg table
metadata).
2. When enabled, resolve the effective HDFS user per table (owner) and route:
- AMS optimizing commits (metadata avro, snapshot commit, delete files)
- Any AMS-side file cleanup tied to optimizing
- Optimizer task execution that writes replacement files (if tasks also
run as service principal today)
through `TableMetaStore.doAsImpersonating(owner, ...)`.
3. **Document** required Hadoop `hadoop.proxyuser.<catalog-principal>.*`
settings and interaction with Ranger (authorization still applies to the
**effective** user).
4. **Tests**: Kerberos or mini-cluster test with a table owned by user A and
catalog principal B; commit without extra ACLs when impersonation is on.
5. **Safety**: Optional allowlist of owners/groups; clear behavior when
owner is null or `hdfs`; fail fast with a actionable error if proxy is not
configured.
### Why not only Ranger / ACLs?
Granting the service user global or warehouse-wide write access duplicates
Hive/Impala’s **impersonation** model and increases blast radius. Owner
impersonation matches how multi-tenant Hive warehouses are usually operated and
avoids per-table ACL runbooks for every new table.
### References
- Hadoop proxy users:
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html#Configurations
- Amoro Trino doc: `mixed-format.hdfs.impersonation.enabled` (same product
family, different engine)
- Code: `TableMetaStore.doAsImpersonating`, `AuthenticatedHadoopFileIO`,
`MixedFormatHdfsAuthentication`
### Version
Amoro 0.9.x (behavior checked against 0.9 AMS + Iceberg Hive catalog);
likely applies to earlier 0.6+ AMS deployments with Kerberos.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
