mixermt commented on PR #5898:
URL:
https://github.com/apache/datafusion-comet/pull/5898#issuecomment-6548246154
Answers below; each change is its own commit, plus one fix for the fork CI
(the encryption-zone test used the deprecated two-argument
`createEncryptionZone`, which the strict-warnings job rejects).
**1. Identity test** — added.
- `MiniDFSCluster` enforces permissions by default. The check was vacuous
because the JVM's user is the superuser of every NameNode it starts.
- The new test starts a cluster under another user's UGI, so the JVM's user
is an ordinary one: with the data directory at 700 the native scan reads it;
once the directory belongs to someone else, the native read is denied with
`AccessControlException`, as Spark's is.
- `HADOOP_USER_NAME` itself is process environment the client reads at
connect, which a JVM test cannot set; hdfs-native covers that branch.
**2. Driver-only settings** — forwarded.
- The client's config is the executors' XML overridden by what OpenDAL
passes in, which iceberg-rust fills from `hadoop.*`. Comet already let those
through, so far only for nameservices.
- hdfs-native reads few client keys. Now forwarded under `hadoop.` when the
session sets them (not Hadoop's defaults, which would override the executors'
XML), behind the catalog's own: `dfs.client.use.datanode.hostname`,
`dfs.client.socket-timeout`,
`dfs.client.failover.{max.attempts,sleep.base.millis,sleep.max.millis}`.
- Not forwardable: the per-nameservice `dfs.client.failover.*.<ns>` keys
(OpenDAL dials every NameNode list as a nameservice of its own name, so the
client never looks them up) and `dfs.data.transfer.protection` (the client
negotiates SASL whenever it is set; the JVM client only on a secure cluster,
which falls back).
- The docs now list what the client ignores: short-circuit and hedged reads,
`dfs.client.retry.*`, `ipc.client.*`, `hadoop.rpc.protection`.
**3. Probe at scale** — both points right; changed.
- Listings are kept per parent (filesystem, parent → child flags), so
entries scale with parents, not partitions; a directory missing from a kept
listing, a new partition, re-lists only its parent.
- Expiry honors `spark.sql.metadataCacheTTLSeconds`, Spark's TTL for its own
partition file metadata cache (default none). That bounds the
purge-and-recreate case.
- A scan that would take more than 1000 listings is declined with that in
the reason; the listings taken are kept, so later scans converge.
- `listEncryptionZones` is superuser-only and `getEZForPath` is one call per
path, so a listing per parent stays the cheapest honest check for a plain user.
**4. Pin audit** — `b154603a..de0a6ac8`, 59 commits.
- Read path (7): #3437 (id-less nested structs read NULL, asserted here),
#3454 (delete row selection), #3502 (a NaN row-group bound keeps the group),
#3473 (an empty projection reads no data columns; the MOR `COUNT(*)` tests
cover it), #2930 (`arrow.uuid` field metadata; the UUID tests pass), #3111
(HDFS), #3416 (a no-op: Comet already binds `rewrite_not()` output).
- Writer (2): #3539 (NaN bounds skipped; dormant until parquet 60), #3358
(float path formatting, which Comet renders itself). The other 50 are catalogs,
transactions, `TableScan` planning, CI and deps.
- #3483 and #3476 change `TableScan::plan_files` and `RawLiteral` JSON,
neither on Comet's path.
- Position deletes for #3454: the `CometIcebergNativeSuite` MOR tests (v2
position deletes, v3 DVs, long runs and scattered singles, compaction,
partitioned, `_pos`), `CometIcebergRewriteActionSuite`,
`IcebergReadFromS3Suite` and the MOR read-backs in
`CometIcebergWriteActionSuite`; upstream, `TestSparkReaderDeletes` (including
multi-row-group position deletes) and the `spark-extensions` MOR suites run
under Comet. Added one MOR read over HDFS to `CometIcebergHdfsSuite`.
- Pre-existing, pointed at by #3483's notes: page-index pruning on an
int→long or float→double promoted column drops rows with row selection on. That
is #7138 here and apache/iceberg-rust#3482 (#3507 open) upstream, unchanged by
this bump.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
