andygrove opened a new pull request, #5035:
URL: https://github.com/apache/datafusion-comet/pull/5035
## Which issue does this PR close?
There is no dedicated tracking issue; `randstr` was previously listed as
planned (🔜) in the expression support guide.
Closes #.
## Rationale for this change
Spark 4.0+ `randstr(length[, seed])` fell back to Spark because Comet had no
native implementation. Spark generates a random alphanumeric string per row via
`ExpressionImplUtils.randStr`, seeding an `XORShiftRandom` with `seed +
partitionIndex` per partition and drawing `abs(rng.nextInt() % 62)` per
character (mapped onto `0-9`/`a-z`/`A-Z`). Comet already ports that exact
`XORShiftRandom` for native `rand`/`randn`, so `randstr` can run natively and
produce results that match Spark bit for bit, keeping the enclosing operator on
the native pipeline instead of falling back.
## What changes are included in this PR?
This change was scaffolded with the project's `implement-comet-expression`
skill (and audited with `audit-comet-expression`).
- Native `RandStrExpr`
(`native/spark-expr/src/nondetermenistic_funcs/randstr.rs`), modeled on the
`rand`/`uuid` pattern: it keeps a per-partition `XorShiftRandom` in a `Mutex`
so state advances across batches, and encodes each `length`-byte string into a
pre-sized `StringBuilder`. Added `XorShiftRandom::next_i32()` (Java
`Random.nextInt()`).
- Proto `RandStr { length, seed }` message wired through the expression
registry with a `RandStrBuilder` that combines the seed with the partition
index (matching `rand`/`randn`/`shuffle`).
- `CometRandStr` serde emitting the resolved literal length and seed.
Because `RandStr` is a Spark 4.0+ class, the serde lives under `spark-4.x` and
is registered via `Spark4xCometExprShim.sparkVersionSpecificStringExpressions`,
so Spark 3.4/3.5 (which lack the class) still compile.
- Support-status doc flipped from planned to supported, and a `misc_funcs`
audit entry recording the cross-version findings.
The audit confirmed `ExpressionImplUtils.randStr` and the `XORShiftRandom`
seeding are byte-identical across Spark 4.0.1 and 4.1.1, so no intra-4.x shim
is needed. `length` and `seed` are required-foldable; Comet handles
non-negative literal length with a literal seed and otherwise falls back to
Spark (which also raises the negative-length error).
## How are these changes tested?
- Rust unit tests in `randstr.rs`: length and alphanumeric charset, zero
length, determinism per (length, seed), and per-batch state continuity.
- `expressions/string/randstr.sql` (Spark 4.0+, gated with `MinSparkVersion:
4.0`): seeded `randstr(length, seed)` asserts bit-for-bit equality with Spark
across single and multiple rows, negative/zero/boundary seeds, lengths
0/1/12/64, and combined with other expressions; a negative length asserts the
`INVALID_PARAMETER_VALUE.LENGTH` fallback; the no-seed `randstr(length)` form
checks deterministic length and alphabet properties. Verified natively on Spark
4.1.2 (default); on Spark 3.5 the build compiles and the gated file is skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
