obarisk opened a new issue, #53525:
URL: https://github.com/apache/airflow/issues/53525
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
16.1.0
### Apache Airflow version
3.0.3
### Operating System
debian 12
### Deployment
Other Docker-based deployment
### Deployment details
podman pod (similar to docker-compose).
### What happened
when using
`airflow.providers.google.cloud.transfers.gcs_to_bigquery.GCSToBigQueryOperator`
the task will fail during xcom_push
note, the target table will be created. the data is already loaded.
but the task just failed.
### What you think should happen instead
there shouldn't be errors.
### How to reproduce
using an example dag
```python
from airflow.sdk import DAG
from airflow.providers.google.cloud.transfers.gcs_to_bigquery import (
GCSToBigQueryOperator
)
dag = DAG(dag_id="gcs_to_bigquery", tags=["bug"])
gcs_to_bigquery = GCSToBigQueryOperator(
task_id="gcs_to_bigquery",
gcp_conn_id="<a google connection>",
bucket="<a bucket>",
source_objects=["<< path to a/some parquets"],
destination_project_dataset_table="<< some dataset >>.__test__bug01",
source_format="PARQUET",
write_disposition="WRITE_TRUNCATE",
dag=dag)
```
the errors
> Task failed with exception: source="task"
> TypeError: cannot serialize object of type <class 'type'>
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py",
line 877 in run
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py",
line 1164 in _execute_task
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py",
line 397 in wrapper
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py",
line 439 in execute
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/links/base.py",
line 81 in persist
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py",
line 410 in xcom_push
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py",
line 533 in _xcom_push
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/xcom.py",
line 71 in set
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/xcom.py",
line 329 in serialize_value
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/serialization/serde.py",
line 135 in serialize
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/serialization/serde.py",
line 175 in serialize
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/serialization/serde.py",
line 135 in serialize
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/serialization/serde.py",
line 182 in serialize
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/serialization/serde.py",
line 135 in serialize
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/serialization/serde.py",
line 182 in serialize
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/serialization/serde.py",
line 135 in serialize
> File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/serialization/serde.py",
line 185 in serialize
### Anything else
it's related to this.
`BigQueryTableLink.persist(**persist_kwargs)` (in `gcs_to_bigquery.py`)
```python
persist_kwargs = {
"context": context,
"task_instance": self,
"table_id": table,
}
```
self (GCSToBigQueryOperator(BaseOperator)) can't be serialized by xcom.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
