hcrosse opened a new issue, #2416:
URL: https://github.com/apache/iceberg-go/issues/2416
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
A `StartsWith` or `NotStartsWith` row filter fails at read time with `not
implemented: starts_with`. Prefix filters can't be used on an unpartitioned
table, or on a partitioned table unless the partition values decide the
predicate.
I reproduced this on main at `8ef0a092` and on v0.7.0. Both use arrow-go
v18.8.0.
**Reproduction**
This program writes three rows to an unpartitioned table in a local Hadoop
catalog and filters `name` by the prefix `an`:
```go
package main
import (
"context"
"fmt"
"log"
"os"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/apache/iceberg-go"
"github.com/apache/iceberg-go/catalog"
_ "github.com/apache/iceberg-go/catalog/hadoop"
"github.com/apache/iceberg-go/table"
)
func main() {
ctx := context.Background()
dir, err := os.MkdirTemp("", "warehouse")
check(err)
cat, err := catalog.Load(ctx, "local", iceberg.Properties{"type":
"hadoop", "warehouse": dir})
check(err)
check(cat.CreateNamespace(ctx, catalog.ToIdentifier("db"), nil))
tbl, err := cat.CreateTable(ctx, catalog.ToIdentifier("db", "t"),
iceberg.NewSchema(0,
iceberg.NestedField{ID: 1, Name: "name", Type:
iceberg.PrimitiveTypes.String}))
check(err)
sc := arrow.NewSchema([]arrow.Field{{Name: "name", Type:
arrow.BinaryTypes.String, Nullable: true}}, nil)
data, err := array.TableFromJSON(memory.DefaultAllocator, sc,
[]string{`[{"name": "anna"}, {"name": "bob"}, {"name": "andy"}]`})
check(err)
tbl, err = tbl.AppendTable(ctx, data, 1024, nil)
check(err)
fmt.Println("expected: rows=2")
result, err :=
tbl.Scan(table.WithRowFilter(iceberg.StartsWith(iceberg.Reference("name"),
"an"))).ToArrowTable(ctx)
if err != nil {
fmt.Println("actual: error:", err)
} else {
fmt.Printf("actual: rows=%d\n", result.NumRows())
}
}
func check(err error) {
if err != nil {
log.Fatal(err)
}
}
```
```sh
go mod init repro && go get github.com/apache/iceberg-go@8ef0a092
go mod tidy && go run .
```
Output at main and at v0.7.0:
```text
expected: rows=2
actual: error: not implemented: starts_with
```
The filter should return `anna` and `andy`.
**Where it fails**
I also ran a larger program with four rows and three partition specs.
`NotStartsWith` failed the same way, and so did `ReadTasks`. `ToArrowTable`
reads through `ToArrowRecords`, which calls `ReadTasks`.
The error appears whenever partition values don't decide the predicate:
- On an unpartitioned table.
- On a column that is not a partition source. `StartsWith(name, "an")`
failed on a table partitioned by `identity(region)`.
- On a `truncate[W]` column with a prefix longer than `W`. Under
`truncate[2](name)`, `StartsWith(name, "ann")` failed.
Scans succeeded when planning removed the predicate. Under
`identity(region)`, `StartsWith(region, "us")` returned two rows. Under
`truncate[2](name)`, both `an` predicates returned two rows each.
**Cause**
iceberg-go translates both predicates to the Substrait function
`starts_with`. arrow-go v18.8.0 has no mapping for that function in its
expression executor and no compute kernel for it. The existing prefix tests
check translation, planning and pruning, but none read rows with a prefix
predicate still to evaluate.
**Related links**
-
https://github.com/apache/iceberg-go/blob/8ef0a092fcbaf70f99aa8935481bc95071eb41b1/table/substrait/substrait.go#L480-L487
converts both predicates to `starts_with`.
-
https://github.com/apache/arrow-go/blob/72e9a5695337bab9ac01ec1e4d58177bc4949dc3/arrow/compute/exprs/exec.go#L604-L606
returns the error for an unmapped function.
- https://github.com/apache/arrow-go/issues/1329 is an open request for
string kernels in arrow-go compute, including `starts_with`. It does not
mention iceberg-go.
**Not checked**
I did not test binary prefix filters through the public API, partition
transforms other than identity and truncate, or remote catalogs and storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
