alexburgoon opened a new issue, #32342: URL: https://github.com/apache/superset/issues/32342
### Bug description When upgrading on Kubernetes from version 1.5.3 to 4.1.1, I run the `superset db upgrade` command in an init job and receive a `ERROR [flask_migrate] Error: maximum recursion depth exceeded while calling a Python object` during migration `a9422eeaae74, new_dataset_models_take_2`. This appears to happen when the number of datasets stored in the metadata database is around 200. It worked with just a few datasets. Reproduction Steps: 1. Deploy Superset version 1.5.3 on Kubernetes 2. Import 200 datasets 3. Deploy upgraded Docker image version 4.1.1 4. Run `superset db upgrade` Expected Result: Database migrations run successfully. ### Screenshots/recordings _No response_ ### Superset version 4.1.1 ### Python version 3.11 ### Node version Not applicable ### Browser Not applicable ### Additional context Logs from init script: ``` [notice] A new release of pip is available: 25.0 -> 25.0.1 [notice] To update, run: pip install --upgrade pip ERROR: You must give at least one requirement to install (see "pip help install") Upgrading DB schema... Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py] 2025-02-21 11:39:15,227:INFO:superset.initialization:Setting database isolation level to READ COMMITTED /usr/local/lib/python3.10/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. warnings.warn( INFO [alembic.env] Starting the migration scripts. INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running upgrade 58df9d617f14 -> 2ed890b36b94, rm_time_range_endpoints_from_qc INFO [alembic.runtime.migration] Running upgrade 2ed890b36b94 -> b0d0249074e4, deprecate time_range_endpoints v2 INFO [alembic.runtime.migration] Running upgrade 2ed890b36b94 -> 8b841273bec3, sql_lab_models_database_constraint_updates INFO [alembic.runtime.migration] Running upgrade 8b841273bec3, b0d0249074e4 -> 9d8a8d575284, merge point INFO [alembic.runtime.migration] Running upgrade 9d8a8d575284 -> cecc6bf46990, rm_time_range_endpoints_2 INFO [alembic.runtime.migration] Running upgrade cecc6bf46990 -> ad07e4fdbaba, rm_time_range_endpoints_from_qc_3 INFO [alembic.runtime.migration] Running upgrade ad07e4fdbaba -> a9422eeaae74, new_dataset_models_take_2 ERROR [flask_migrate] Error: maximum recursion depth exceeded while calling a Python object slices updated with no time_range_endpoints: 0 >> Copy 10 physical tables to sl_tables... >> Copy 202 SqlaTable to sl_datasets... Copy dataset owners... Link physical datasets with tables... >> Copy 2,491 table columns to sl_columns... Link all columns to sl_datasets... >> Copy 207 metrics to sl_columns... Link metric columns to datasets... >> Run postprocessing on 2,698 columns [Column 1 to 2,698] 2,493 may be updated Assign table column relations... >> Run postprocessing on 202 datasets Process dataset 1~202... Initializing roles... Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py] 2025-02-21 11:39:25,436:INFO:superset.initialization:Setting database isolation level to READ COMMITTED /usr/local/lib/python3.10/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. warnings.warn( 2025-02-21 11:39:29,676:INFO:superset.security.manager:Syncing role definition 2025-02-21 11:39:29,811:INFO:superset.security.manager:Syncing Admin perms 2025-02-21 11:39:29,821:INFO:superset.security.manager:Syncing Alpha perms 2025-02-21 11:39:29,841:INFO:superset.security.manager:Syncing Gamma perms 2025-02-21 11:39:29,878:INFO:superset.security.manager:Syncing sql_lab perms 2025-02-21 11:39:29,928:INFO:superset.security.manager:Fetching a set of all perms to lookup which ones are missing 2025-02-21 11:39:29,953:INFO:superset.security.manager:Creating missing datasource permissions. Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context self.dialect.do_execute( File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute cursor.execute(statement, parameters) psycopg2.errors.UndefinedColumn: column tables.catalog_perm does not exist LINE 1: ...s_perm, tables.schema_perm AS tables_schema_perm, tables.cat... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/app/superset/utils/decorators.py", line 259, in wrapped result = func(*args, **kwargs) File "/app/superset/cli/main.py", line 68, in init security_manager.sync_role_definitions() File "/app/superset/security/manager.py", line 1070, in sync_role_definitions self.create_missing_perms() File "/app/superset/security/manager.py", line 1018, in create_missing_perms datasources = SqlaTable.get_all_datasources() File "/app/superset/connectors/sqla/models.py", line 1955, in get_all_datasources return qry.all() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2773, in all return self._iter().all() File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter result = self.session.execute( File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1717, in execute result = conn._execute_20(statement, params or {}, execution_options) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20 return meth(self, args_10style, kwargs_10style, execution_options) File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection return connection._execute_clauseelement( File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement ret = self._execute_context( File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context self._handle_dbapi_exception( File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception util.raise_( File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_ raise exception File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context self.dialect.do_execute( File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column tables.catalog_perm does not exist LINE 1: ...s_perm, tables.schema_perm AS tables_schema_perm, tables.cat... ^ [SQL: SELECT tables.uuid AS tables_uuid, tables.created_on AS tables_created_on, tables.changed_on AS tables_changed_on, tables.id AS tables_id, tables.description AS tables_description, tables.default_endpoint AS tables_default_endpoint, tables.is_featured AS tables_is_featured, tables.filter_select_enabled AS tables_filter_select_enabled, tables."offset" AS tables_offset, tables.cache_timeout AS tables_cache_timeout, tables.params AS tables_params, tables.perm AS tables_perm, tables.schema_perm AS tables_schema_perm, tables.catalog_perm AS tables_catalog_perm, tables.is_managed_externally AS tables_is_managed_externally, tables.external_url AS tables_external_url, tables.table_name AS tables_table_name, tables.main_dttm_col AS tables_main_dttm_col, tables.database_id AS tables_database_id, tables.fetch_values_predicate AS tables_fetch_values_predicate, tables.schema AS tables_schema, tables.catalog AS tables_catalog, tables.sql AS tables_sql, tables.is_sqllab_view AS tables_is_s qllab_view, tables.template_params AS tables_template_params, tables.extra AS tables_extra, tables.normalize_columns AS tables_normalize_columns, tables.always_filter_main_dttm AS tables_always_filter_main_dttm, tables.created_by_fk AS tables_created_by_fk, tables.changed_by_fk AS tables_changed_by_fk FROM tables WHERE tables.is_sqllab_view = false] (Background on this error at: https://sqlalche.me/e/14/f405) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/bin/superset", line 8, in <module> sys.exit(superset()) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 358, in decorator return __ctx.invoke(f, *args, **kwargs) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 358, in decorator return __ctx.invoke(f, *args, **kwargs) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/app/superset/utils/decorators.py", line 266, in wrapped return on_error(ex) File "/app/superset/utils/decorators.py", line 234, in on_error raise reraise() from ex sqlalchemy.exc.SQLAlchemyError: () ``` ### Checklist - [x] I have searched Superset docs and Slack and didn't find a solution to my problem. - [x] I have searched the GitHub issue tracker and didn't find a similar bug report. - [x] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
