drummerwolli opened a new pull request, #34844: URL: https://github.com/apache/superset/pull/34844
This commit implements the dataset model changes proposed in GitHub issue #14909 (SIP-68) to create a better semantic layer foundation for Apache Superset. ## New Models Added: - **Table**: Represents physical tables/views in databases - **Dataset**: Represents both physical and virtual datasets with semantic metadata - **Column**: Unified model for table columns, metrics, and derived columns ## Key Features: - **Shadow Writing**: Keeps old SqlaTable and new models in sync during transition - **Feature Flag Control**: Configurable via DATASET_SHADOW_WRITING_ENABLED - **Full Backward Compatibility**: Existing APIs and functionality unchanged - **Database Migrations**: Includes schema creation and data population scripts ## Implementation Details: - New models follow SIP-68 specification exactly - Proper indexing, constraints, and relationships established - Handles both physical and virtual datasets correctly - PostgreSQL reserved keyword conflicts resolved ## Testing: - Comprehensive unit tests included - Shadow writing validated with Docker Compose environment - Successfully creates and syncs data between old and new models - Application starts and runs normally with new models integrated ## Files Changed: - superset/models/dataset.py - New SIP-68 dataset models - superset/datasets/shadow_writer.py - Shadow writing implementation - superset/migrations/ - Database migration scripts - superset/config.py - Feature flag configuration - superset/connectors/sqla/models.py - Event hooks for shadow writing - tests/unit_tests/models/test_dataset_models.py - Comprehensive tests This provides a clean foundation for migrating Superset's semantic layer while maintaining production stability through gradual, feature-flagged rollout. 🤖 Generated with [Claude Code](https://claude.ai/code) <!--- Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/ Example: fix(dashboard): load charts correctly --> ### SUMMARY <!--- Describe the change below, including rationale and design decisions --> ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF <!--- Skip this if not applicable --> ### TESTING INSTRUCTIONS <!--- Required! What steps can be taken to manually verify the changes? --> ### ADDITIONAL INFORMATION <!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue --> - [ ] Has associated issue: - [ ] Required feature flags: - [ ] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [ ] Introduces new feature or API - [ ] Removes existing feature or API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
