hcrosse opened a new issue, #2425:
URL: https://github.com/apache/iceberg-go/issues/2425
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
A row filter fails when the filtered column has a dictionary type in the
Parquet file's embedded Arrow schema. PyIceberg writes such files by default
from a pandas `Categorical` column, so iceberg-go can't filter on that column
in a table PyIceberg wrote from a DataFrame.
I reproduced this on main at `8ef0a092` and on v0.7.0. Both use arrow-go
v18.8.0. The tables were written with PyIceberg 0.12.0, pyarrow 26.0.1 and
pandas 3.1.0, using default write options and a local SQLite catalog.
**Reproduction**
This script appends three rows from a pandas `Categorical` column and prints
the type stored in the file:
```python
import os
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from pyiceberg.catalog.sql import SqlCatalog
WAREHOUSE = "/tmp/iceberg-dict-filter"
os.makedirs(WAREHOUSE)
cat = SqlCatalog("default", uri=f"sqlite:///{WAREHOUSE}/catalog.db",
warehouse=f"file://{WAREHOUSE}")
cat.create_namespace("db")
df = pd.DataFrame({"id": [1, 2, 3], "s": pd.Categorical(["a", "b", "a"])})
data = pa.Table.from_pandas(df, preserve_index=False)
tbl = cat.create_table("db.t", schema=data.schema)
tbl.append(data)
path = tbl.inspect.files()["file_path"][0].as_py().removeprefix("file://")
print("stored s:", pq.read_schema(path).field("s").type)
```
This program loads the same table and filters on `s = "a"`:
```go
package main
import (
"context"
"fmt"
"log"
"github.com/apache/iceberg-go"
"github.com/apache/iceberg-go/catalog"
sqlcat "github.com/apache/iceberg-go/catalog/sql"
"github.com/apache/iceberg-go/table"
"github.com/uptrace/bun/driver/sqliteshim"
)
const warehouse = "/tmp/iceberg-dict-filter"
func main() {
ctx := context.Background()
cat, err := catalog.Load(ctx, "default", iceberg.Properties{
"uri": "file:" + warehouse + "/catalog.db",
"type": "sql",
sqlcat.DriverKey: sqliteshim.ShimName,
sqlcat.DialectKey: string(sqlcat.SQLite),
})
check(err)
tbl, err := cat.LoadTable(ctx, catalog.ToIdentifier("db", "t"))
check(err)
fmt.Println("expected: rows=2")
result, err :=
tbl.Scan(table.WithRowFilter(iceberg.EqualTo(iceberg.Reference("s"),
"a"))).ToArrowTable(ctx)
if err != nil {
fmt.Println("actual: error:", err)
} else {
fmt.Printf("actual: rows=%d\n", result.NumRows())
}
}
func check(err error) {
if err != nil {
log.Fatal(err)
}
}
```
Save the script as `setup.py` and the program as `main.go` in an empty
directory, then run the commands below. Delete `/tmp/iceberg-dict-filter`
before running them again.
```sh
pip install "pyiceberg[sql-sqlite,pyarrow]==0.12.0" pandas && python setup.py
go mod init repro && go get github.com/apache/iceberg-go@8ef0a092
go mod tidy && go run .
```
Setup output:
```text
stored s: dictionary<values=string, indices=int8, ordered=0>
```
Go output at main and at v0.7.0:
```text
expected: rows=2
actual: error: invalid: referenced field %!s(<nil>) was
dictionary<values=utf8, indices=int8, ordered=false>, but should have been utf8
```
The filter should return the two rows where `s` is `"a"`.
**Other cases**
Two other PyIceberg paths store the same type and fail the same way, with
`int32` indices:
- `append()` of a pyarrow table whose string column uses
`dictionary_encode()`.
- `pq.write_table()` of that table, then `add_files()`.
The problem is the embedded Arrow type, not Parquet's dictionary page
encoding. On all three tables, an unfiltered scan and a filter on another
column succeeded. A table with plain string arrays returned two rows for the
same filter. iceberg-go's own `Append` stores dictionary input as `utf8`, and
that table filters correctly too.
**Cause**
The scan evaluates the filter on the file batch before projecting the
columns. arrow-go's expression executor then rejects the column because the
expression expects `utf8`. That check accepts `large_utf8` for a string
expression, but not a dictionary whose values are `utf8`.
The reader should evaluate the predicate against the column's string values.
arrow-go's comparison kernels already decode dictionary inputs
(https://github.com/apache/arrow-go/blob/72e9a5695337bab9ac01ec1e4d58177bc4949dc3/arrow/compute/scalar_compare.go#L56),
so only the field-reference check stops this one. The type difference could be
handled there, as https://github.com/apache/arrow-go/pull/440 did for
`large_utf8`, or by decoding the column in iceberg-go before the filter. I have
not tested either change.
**Related links**
- https://github.com/apache/iceberg-go/issues/486 is the closed report of
the same error for `large_utf8`.
-
https://github.com/apache/arrow-go/blob/72e9a5695337bab9ac01ec1e4d58177bc4949dc3/arrow/compute/exprs/exec.go#L525-L536
contains the type check that rejects this batch.
-
https://github.com/apache/iceberg-go/blob/8ef0a092fcbaf70f99aa8935481bc95071eb41b1/table/arrow_scanner.go#L2839-L2863
adds the filter to the pipeline before the projection step.
**Not checked**
I did not test files written by Spark, Java parquet-mr, arrow-rs, Polars or
DuckDB, or remote catalogs and storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
