stahh opened a new issue, #38744:
URL: https://github.com/apache/arrow/issues/38744
### Describe the bug, including details regarding any error messages,
version, and platform.
Error when tried to upload parquet files that imported from Snowflake into
Postgres
```
Start process
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value:
TypeNotSupported { field: "NUMERIC_FIELD", tp: Decimal128(38, 0), msg: "unknown
type" }', py/src/lib.rs:35:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "/home/.../load_parquet.py", line 24, in <module>
parquet_to_pg()
File "/home/.../load_parquet.py", line 10, in process_hem2bito
encoder = ArrowToPostgresBinaryEncoder(dataset.schema)
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value:
TypeNotSupported { field: "NUMERIC_FIELD", tp: Decimal128(38, 0), msg: "unknown
type" }
```
Process finished with exit code 1
-----------------------------------------------------------------------------------------------------------------
Step to reproduce:
1.
```sql
create or replace TABLE DB.SCHEMA.TBL (
VARCHAR_FIELD VARCHAR(256),
NUMERIC_FIELD NUMBER(38,0));
```
2.
```sql
insert into TABLE DB.SCHEMA.TBL (VARCHAR_FIELD, NUMERIC_FIELD) VALUES
('string 1', -2395872395873952873), ('string_2', 398537985637496873496);```
3.
```sql
COPY INTO 's3://test_buckettest/'
FROM DB.SCHEMA.TBL
credentials = (
aws_key_id='**********************'
aws_secret_key='********************************'
)
header=True
OVERWRITE = TRUE
file_format = (
type = PARQUET,
COMPRESSION = NONE
);
```
4. Code to run
```python
import psycopg
import pyarrow.dataset as ds
from pgpq import ArrowToPostgresBinaryEncoder
print('Start process')
s3_uri = f's3://test_bucket/test/'
dataset = ds.dataset(s3_uri, format="parquet")
encoder = ArrowToPostgresBinaryEncoder(dataset.schema)
with psycopg.connect('postgresql://**********************') as conn:
with conn.cursor() as cursor:
with cursor.copy(
"COPY public.companies FROM STDIN WITH (FORMAT BINARY)") as
copy:
copy.write(encoder.write_header())
for batch in dataset.to_batches():
copy.write(encoder.write_batch(batch))
copy.write(encoder.finish())
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
