valentinsoe opened a new issue, #7385:
URL: https://github.com/apache/datafusion-comet/issues/7385
### Describe the bug
Since #4413 made `parse_url` Compatible, `CometParseUrl` /
`CometTryParseUrl` run natively by default. For any input without a scheme, the
native implementation returns NULL for every part, including `QUERY` with a
key. Spark parses such inputs with `java.net.URI`, which accepts relative
references (RFC 3986 section 4.2), and returns their path, query and fragment.
The branch responsible, in `native/spark-expr/src/url_funcs/parse_url.rs`
(tag 1.1.0, L146-151, still present on `main`):
```rust
if scheme.is_none() {
if value.contains("://") {
return Err(invalid_url_err(value));
}
return Ok(None);
}
```
https://github.com/apache/datafusion-comet/blob/1.1.0/native/spark-expr/src/url_funcs/parse_url.rs#L146-L151
With `failOnError = false` (the default, non-ANSI) the result is silently
NULL. No error or fallback is raised, so queries succeed with wrong results.
### Steps to reproduce
Local mode, Comet 1.1.0 from Maven Central (`comet-spark-spark4.1_2.13`,
`comet-spark-spark4.0_2.13`, `comet-spark-spark3.5_2.12`), JDK 17:
```python
from pyspark.sql import SparkSession
spark = (SparkSession.builder.master("local[2]")
.config("spark.jars.packages",
"org.apache.datafusion:comet-spark-spark4.1_2.13:1.1.0")
.config("spark.plugins", "org.apache.spark.CometPlugin")
.config("spark.shuffle.manager",
"org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager")
.config("spark.memory.offHeap.enabled", "true")
.config("spark.memory.offHeap.size", "2g")
.getOrCreate())
urls = ["/shop/item/123?ref=abc&id=42#top", "//example.com/shop?id=7",
"?id=7",
"https://example.com/shop/item/123?ref=abc&id=42";]
spark.createDataFrame([(u,) for u in urls], "u
string").write.mode("overwrite").parquet("/tmp/t")
for enabled in ["true", "false"]:
spark.conf.set("spark.comet.enabled", enabled)
df = spark.read.parquet("/tmp/t").selectExpr(
"u", "parse_url(u, 'QUERY', 'id')", "parse_url(u, 'QUERY')",
"parse_url(u, 'PATH')", "parse_url(u, 'REF')", "parse_url(u,
'HOST')")
df.explain()
df.show(truncate=False)
```
With `spark.comet.enabled=true` the plan is `CometProject` over
`CometNativeScan`. Results are identical on Spark 4.1.1, 4.0.1 and 3.5.3:
| input | part | Spark (Comet disabled) | Comet 1.1.0 native |
|---|---|---|---|
| `/shop/item/123?ref=abc&id=42#top` | QUERY, 'id' | `42` | NULL |
| | QUERY | `ref=abc&id=42` | NULL |
| | PATH | `/shop/item/123` | NULL |
| | REF | `top` | NULL |
| `//example.com/shop?id=7` | QUERY, 'id' | `7` | NULL |
| | PATH | `/shop` | NULL |
| | HOST | `example.com` | NULL |
| `?id=7` | QUERY, 'id' | `7` | NULL |
| | PATH | `''` | NULL |
| `https://example.com/shop/item/123?ref=abc&id=42` | all parts | | same as
Spark |
### Expected behavior
The same results as Spark: for a relative reference, PROTOCOL, HOST,
AUTHORITY, USERINFO and PORT are NULL, while PATH, QUERY (with or without key),
REF and FILE are taken from the reference, as `java.net.URI` does. For a
network-path reference (`//host/...`), HOST and AUTHORITY are set too.
Until that is implemented, scheme-less inputs should fall back to Spark (or
ParseUrl should be marked Incompatible again) rather than return NULL.
### Additional context
- Introduced by #4413 (b68c6152744, "feat: make parse_url compatible"),
which was verified against Spark 4.0/4.1 using absolute URLs only. Its only
scheme-less test is `test_malformed_no_scheme` (`parse_url("notaurl", "HOST")`
returns NULL), which matches Spark only because HOST is NULL there too.
`parse_url.sql` has no relative URLs. The docs list `parse_url` as fully native
with no notes.
- Related, but a different direction: #7330 covers inputs that
`java.net.URI` rejects while Comet accepts them. This issue is about valid
relative URIs that Comet rejects.
- One query can give inconsistent results. Depending on the plan, Spark can
keep a predicate such as `isnotnull(parse_url(u, 'QUERY', 'id'))` in a JVM
`Filter` while the same expression in the projection becomes a `CometProject`.
Rows then pass the filter but carry NULL in the projected column.
- Workaround: `spark.comet.expression.ParseUrl.enabled=false`, or disable
Comet for the query.
- Versions: Comet 1.1.0; Spark 4.1.1, 4.0.1 and 3.5.3; JDK 17.
### Willingness to contribute
I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
