andygrove commented on PR #5031:
URL:
https://github.com/apache/datafusion-comet/pull/5031#issuecomment-5074714349
Coverage report generated by `CometExpressionCoverageSuite` against the
Spark 4.1 profile:
# Comet expression coverage report (Spark 4.1.2)
Probed 524 registered functions using their documented examples. Rerun under
each Maven Spark profile (`-Pspark-3.4`, `-Pspark-3.5`, `-Pspark-4.0`,
`-Pspark-4.1`) to compare coverage across versions.
| Status | Count |
| --- | --- |
| NATIVE | 330 |
| INCOMPATIBLE | 6 |
| LITERAL_PROBE | 4 |
| GAP | 42 |
| SKIP | 136 |
| ERROR | 6 |
## Scalar gaps (Project fallback) - 38
Actionable candidates: scalar expressions Comet did not convert.
| Function | Expression class | Fallback reason | Probed query |
| --- | --- | --- | --- |
| `assert_true` | `org.apache.spark.sql.catalyst.expressions.AssertTrue` |
raise_error is not supported | `SELECT assert_true(0 < 1) FROM
comet_coverage_probe` |
| `bitmap_bit_position` |
`org.apache.spark.sql.catalyst.expressions.BitmapBitPosition` | Static invoke
expression: bitmapBitPosition is not supported | `SELECT bitmap_bit_position(1)
FROM comet_coverage_probe` |
| `bitmap_bucket_number` |
`org.apache.spark.sql.catalyst.expressions.BitmapBucketNumber` | Static invoke
expression: bitmapBucketNumber is not supported | `SELECT
bitmap_bucket_number(123) FROM comet_coverage_probe` |
| `bitmap_count` | `org.apache.spark.sql.catalyst.expressions.BitmapCount` |
Static invoke expression: bitmapCount is not supported | `SELECT bitmap_count(X
'1010') FROM comet_coverage_probe` |
| `e` | `org.apache.spark.sql.catalyst.expressions.EulerNumber` | E is not
supported | `SELECT e() FROM comet_coverage_probe` |
| `encode` | `org.apache.spark.sql.catalyst.expressions.Encode` | Static
invoke expression: encode is not supported | `SELECT encode('abc', 'utf-8')
FROM comet_coverage_probe` |
| `input_file_block_length` |
`org.apache.spark.sql.catalyst.expressions.InputFileBlockLength` | Native
DataFusion scan is not compatible with input_file_name, input_file_block_start,
or input_file_block_length | `SELECT input_file_block_length() FROM
comet_coverage_probe` |
| `input_file_block_start` |
`org.apache.spark.sql.catalyst.expressions.InputFileBlockStart` | Native
DataFusion scan is not compatible with input_file_name, input_file_block_start,
or input_file_block_length | `SELECT input_file_block_start() FROM
comet_coverage_probe` |
| `input_file_name` |
`org.apache.spark.sql.catalyst.expressions.InputFileName` | Native DataFusion
scan is not compatible with input_file_name, input_file_block_start, or
input_file_block_length | `SELECT input_file_name() FROM comet_coverage_probe` |
| `is_valid_utf8` | `org.apache.spark.sql.catalyst.expressions.IsValidUTF8`
| invoke is not supported | `SELECT is_valid_utf8('Spark') FROM
comet_coverage_probe` |
| `is_variant_null` |
`org.apache.spark.sql.catalyst.expressions.variant.IsVariantNull` | Static
invoke expression: isVariantNull is not supported | `SELECT
is_variant_null(parse_json('null')) FROM comet_coverage_probe` |
| `java_method` |
`org.apache.spark.sql.catalyst.expressions.CallMethodViaReflection` |
java_method is not supported | `SELECT java_method('java.util.UUID',
'randomUUID') FROM comet_coverage_probe` |
| `make_interval` | `org.apache.spark.sql.catalyst.expressions.MakeInterval`
| make_interval is not supported | `SELECT make_interval(100, 11, 1, 1, 12, 30,
01.001001) FROM comet_coverage_probe` |
| `make_valid_utf8` |
`org.apache.spark.sql.catalyst.expressions.MakeValidUTF8` | invoke is not
supported | `SELECT make_valid_utf8('Spark') FROM comet_coverage_probe` |
| `parse_json` |
`org.apache.spark.sql.catalyst.expressions.variant.ParseJsonExpressionBuilder`
| Static invoke expression: parseJson is not supported | `SELECT
parse_json('{"a":1,"b":0.8}') FROM comet_coverage_probe` |
| `quote` | `org.apache.spark.sql.catalyst.expressions.Quote` | Static
invoke expression: quote is not supported | `SELECT quote('Don\'t') FROM
comet_coverage_probe` |
| `raise_error` |
`org.apache.spark.sql.catalyst.expressions.RaiseErrorExpressionBuilder` |
raise_error is not supported | `SELECT raise_error('custom error message') FROM
comet_coverage_probe` |
| `randstr` | `org.apache.spark.sql.catalyst.expressions.RandStr` | randstr
is not supported | `SELECT randstr(3, 0) AS result FROM comet_coverage_probe` |
| `reflect` |
`org.apache.spark.sql.catalyst.expressions.CallMethodViaReflection` | reflect
is not supported | `SELECT reflect('java.util.UUID', 'randomUUID') FROM
comet_coverage_probe` |
| `schema_of_variant` |
`org.apache.spark.sql.catalyst.expressions.variant.SchemaOfVariant` | Static
invoke expression: schemaOfVariant is not supported | `SELECT
schema_of_variant(parse_json('null')) FROM comet_coverage_probe` |
| `sentences` | `org.apache.spark.sql.catalyst.expressions.Sentences` |
Static invoke expression: getSentences is not supported | `SELECT sentences('Hi
there! Good morning.') FROM comet_coverage_probe` |
| `st_asbinary` | `org.apache.spark.sql.catalyst.expressions.st.ST_AsBinary`
| Static invoke expression: stAsBinary is not supported | `SELECT
hex(st_asbinary(st_geogfromwkb(X'0101000000000000000000F03F0000000000000040')))
FROM comet_coverage_probe` |
| `st_geogfromwkb` |
`org.apache.spark.sql.catalyst.expressions.st.ST_GeogFromWKB` | Static invoke
expression: stAsBinary is not supported | `SELECT
hex(st_asbinary(st_geogfromwkb(X'0101000000000000000000F03F0000000000000040')))
FROM comet_coverage_probe` |
| `st_geomfromwkb` |
`org.apache.spark.sql.catalyst.expressions.st.ST_GeomFromWKB` | Static invoke
expression: stAsBinary is not supported | `SELECT
hex(st_asbinary(st_geomfromwkb(X'0101000000000000000000F03F0000000000000040')))
FROM comet_coverage_probe` |
| `st_setsrid` | `org.apache.spark.sql.catalyst.expressions.st.ST_SetSrid` |
Static invoke expression: stSrid is not supported | `SELECT
st_srid(st_setsrid(ST_GeogFromWKB(X'0101000000000000000000F03F0000000000000040'),
4326)) FROM comet_coverage_probe` |
| `st_srid` | `org.apache.spark.sql.catalyst.expressions.st.ST_Srid` |
Static invoke expression: stSrid is not supported | `SELECT
st_srid(st_geogfromwkb(X'0101000000000000000000F03F0000000000000040')) FROM
comet_coverage_probe` |
| `to_binary` | `org.apache.spark.sql.catalyst.expressions.ToBinary` |
Static invoke expression: encode is not supported | `SELECT to_binary('abc',
'utf-8') FROM comet_coverage_probe` |
| `to_variant_object` |
`org.apache.spark.sql.catalyst.expressions.variant.ToVariantObject` |
to_variant_object is not supported | `SELECT
to_variant_object(named_struct('a', 1, 'b', 2)) FROM comet_coverage_probe` |
| `try_make_interval` |
`org.apache.spark.sql.catalyst.expressions.TryMakeInterval` | make_interval is
not supported | `SELECT try_make_interval(100, 11, 1, 1, 12, 30, 01.001001)
FROM comet_coverage_probe` |
| `try_parse_json` |
`org.apache.spark.sql.catalyst.expressions.variant.TryParseJsonExpressionBuilder`
| Static invoke expression: parseJson is not supported | `SELECT
try_parse_json('{"a":1,"b":0.8}') FROM comet_coverage_probe` |
| `try_reflect` | `org.apache.spark.sql.catalyst.expressions.TryReflect` |
reflect is not supported | `SELECT try_reflect('java.util.UUID', 'randomUUID')
FROM comet_coverage_probe` |
| `try_validate_utf8` |
`org.apache.spark.sql.catalyst.expressions.TryValidateUTF8` | Static invoke
expression: tryValidateUTF8String is not supported | `SELECT
try_validate_utf8('Spark') FROM comet_coverage_probe` |
| `try_variant_get` |
`org.apache.spark.sql.catalyst.expressions.variant.TryVariantGetExpressionBuilder`
| try_variant_get is not supported | `SELECT try_variant_get(parse_json('{"a":
1}'), '$.a', 'int') FROM comet_coverage_probe` |
| `typeof` | `org.apache.spark.sql.catalyst.expressions.TypeOf` | typeof is
not supported | `SELECT typeof(1) FROM comet_coverage_probe` |
| `uuid` | `org.apache.spark.sql.catalyst.expressions.Uuid` | uuid is not
supported | `SELECT uuid() FROM comet_coverage_probe` |
| `validate_utf8` | `org.apache.spark.sql.catalyst.expressions.ValidateUTF8`
| Static invoke expression: validateUTF8String is not supported | `SELECT
validate_utf8('Spark') FROM comet_coverage_probe` |
| `variant_get` |
`org.apache.spark.sql.catalyst.expressions.variant.VariantGetExpressionBuilder`
| variant_get is not supported | `SELECT variant_get(parse_json('{"a": 1}'),
'$.a', 'int') FROM comet_coverage_probe` |
| `version` | `org.apache.spark.sql.catalyst.expressions.SparkVersion` |
Static invoke expression: getSparkVersion is not supported | `SELECT version()
FROM comet_coverage_probe` |
## Non-scalar gaps (Generate / Aggregate / Window / etc.) - 4
| Function | Expression class | Operator |
| --- | --- | --- |
| `inline` |
`org.apache.spark.sql.catalyst.expressions.InlineExpressionBuilder` |
`Generate` |
| `inline_outer` |
`org.apache.spark.sql.catalyst.expressions.InlineExpressionBuilder` |
`Generate` |
| `json_tuple` | `org.apache.spark.sql.catalyst.expressions.JsonTuple` |
`Generate` |
| `stack` | `org.apache.spark.sql.catalyst.expressions.Stack` | `Generate` |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
