potiuk commented on code in PR #46891: URL: https://github.com/apache/airflow/pull/46891#discussion_r2212538221
########## airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py: ########## @@ -82,6 +88,7 @@ @pytest.fixture(autouse=True) @provide_session def setup(request, dag_maker, session=None): + clear_db_connections() Review Comment: OK. That's the reason https://github.com/apache/airflow/actions/runs/16332955839/job/46140024142?pr=46891 ```python E sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "connection_conn_id_uq" E DETAIL: Key (conn_id)=(git_default) already exists. E E [SQL: INSERT INTO connection (conn_id, conn_type, description, host, schema, login, password, port, is_encrypted, is_extra_encrypted, extra) VALUES (%(conn_id)s, %(conn_type)s, %(description)s, %(host)s, %(schema)s, %(login)s, %(password)s, %(port)s, %(is_encrypted)s, %(is_extra_encrypted)s, %(extra)s) RETURNING connection.id] E [parameters: {'conn_id': 'git_default', 'conn_type': 'git', 'description': 'default git connection', 'host': 'fakeprotocol://test_host.github.com', 'schema': None, 'login': '', 'password': None, 'port': 8081, 'is_encrypted': False, 'is_extra_encrypted': False, 'extra': None}] E (Background on this error at: https://sqlalche.me/e/14/gkpj) ``` It looks like there is a different behaviour on Python 3.13 and <3.12 when it comes to handling ORM objects added and deleted in different sessions. When you look at the dependencies difference between the two, there is no obvious candidate for it Here is the difference between constraints generated on Airflow 3.12 (left) vs 3.13 (right) in our CI. You can see that we are using latest FAB (because we are not bound by FAB provider limit - and some other providers need FAB - BTW. I will bump FAB to 4.8.0 soon in FAB provider - it was relased last week). but more importantly ```diff 3c3 < # This constraints file was automatically generated on 2025-07-17T00:26:52.105735 --- > # This constraints file was automatically generated on 2025-07-17T00:26:30.706526 13c13 < Authlib==1.3.1 --- > Authlib==1.6.0 16c16 < Flask-AppBuilder==4.6.3 --- > Flask-AppBuilder==4.8.0 22d21 < Flask-Session==0.5.0 24c23 < Flask==2.2.5 --- > Flask==2.3.3 36a36 > PyYAML-ft==8.0.0 45c45 < Werkzeug==2.2.3 --- > Werkzeug==3.1.3 53d52 < aiohttp-cors==0.8.1 79d77 < apache-beam==2.66.0 135d132 < cachelib==0.13.0 153d149 < clickclick==20.10.2 157d152 < colorful==0.5.7 161d155 < connexion==2.14.2 166c160 < cryptography==42.0.8 --- > cryptography==45.0.5 188a183 > ecdsa==0.19.1 203d197 < fasteners==0.19 233d226 < google-cloud-bigquery-storage==2.32.0 271c264 < google-genai==1.2.0 --- > google-genai==1.26.0 281,284c274,276 < grpcio-health-checking==1.62.3 < grpcio-status==1.62.3 < grpcio-tools==1.62.3 < grpcio==1.65.5 --- > grpcio-health-checking==1.71.2 > grpcio-status==1.71.2 > grpcio==1.73.1 298c290 < httpx==0.27.0 --- > httpx==0.28.1 313c305 < importlib_metadata==8.4.0 --- > importlib_metadata==8.7.0 339d330 < jsonpickle==3.4.2 386d376 < msgpack==1.1.1 411d400 < objsize==0.7.1 415,416d403 < opencensus-context==0.1.3 < opencensus==0.11.4 422,430c409,417 < opentelemetry-api==1.27.0 < opentelemetry-exporter-otlp-proto-common==1.27.0 < opentelemetry-exporter-otlp-proto-grpc==1.27.0 < opentelemetry-exporter-otlp-proto-http==1.27.0 < opentelemetry-exporter-otlp==1.27.0 < opentelemetry-exporter-prometheus==0.48b0 < opentelemetry-proto==1.27.0 < opentelemetry-sdk==1.27.0 < opentelemetry-semantic-conventions==0.48b0 --- > opentelemetry-api==1.35.0 > opentelemetry-exporter-otlp-proto-common==1.35.0 > opentelemetry-exporter-otlp-proto-grpc==1.35.0 > opentelemetry-exporter-otlp-proto-http==1.35.0 > opentelemetry-exporter-otlp==1.35.0 > opentelemetry-exporter-prometheus==0.56b0 > opentelemetry-proto==1.35.0 > opentelemetry-sdk==1.35.0 > opentelemetry-semantic-conventions==0.56b0 434d420 < orjson==3.11.0 454c440 < pinotdb==5.6.0 --- > pinotdb==5.7.0 471c457 < protobuf==4.25.8 --- > protobuf==5.29.5 478d463 < py-spy==0.4.0 481d465 < pyarrow-hotfix==0.7 491d474 < pydot==1.4.2 530d512 < python3-saml==1.16.0 535d516 < ray==2.47.1 574c555 < sendgrid==6.11.0 --- > sendgrid==6.12.2 584d564 < smart_open==7.3.0.post1 618d597 < starkbank-ecdsa==2.2.0 684c663 < validators==0.34.0 --- > validators==0.35.0 691c670 < weaviate-client==4.9.6 --- > weaviate-client==4.16.0 694c673 < websockets==14.2 --- > websockets==15.0.1 698d676 < xmlsec==1.3.16 701,702d678 < yandex-query-client==0.1.4 < yandexcloud==0.328.0 704,705d679 < ydb-dbapi==0.1.12 < ydb==3.21.6 ``` Currently I can't really reasonably explain why it worked in 3.12 without commits, but it does not work in 3.12 and you have to add commit for such connection added - it looks like the "clear_db_connection" run as last step in the `configure_git_connection_for_dag_bundle` does not really deletes the connection added without commit (left - working in 3.12, not working in 3.13), right - fix that work for both: <img width="1506" height="568" alt="Screenshot 2025-07-17 at 09 26 12" src="https://github.com/user-attachments/assets/3a198e07-d6da-43c1-a03d-52e579089d03"; /> If you look at the implementation of the "clear_db_connection" it uses "create_session" - our sessions are "thread-bound" - so theorethically it should get the same session as the one that was created by "session" fixture in `configure_git_connection_for_dag_bundle` so it **should** be deleting the non-committed connection object created above. So for some reaons (and this is a bit of mistery to me) it seems to get a different session object. The clear_db_connection() in test_dag_run() is added really to avoid any potential side-effects. Those tests rely on connections not having a test git connection created and present - so I additionally (following the pattern that is there) clear the connections there. This was just missing in the test. Let me try it again - this time I will only add clear_db_connection in the tests, but won't "fix" the commit and see if that alone fixes the problem. Then we can either investigate a bit more (it might have some impact on the way we write test) or make decision what to do quickly :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
