adnanhemani commented on code in PR #1600: URL: https://github.com/apache/polaris/pull/1600#discussion_r2096657446
########## site/content/in-dev/unreleased/policy.md: ########## @@ -0,0 +1,207 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Policy +type: docs +weight: 425 +--- + +The Polaris Policy framework empowers organizations to centrally define, manage, and enforce fine-grained governance, lifecycle, and operational rules across all data resources in the catalog. + +With the policy API, you can: +- Create and manage policies +- Attach policies to specific resources (catalogs, namespaces, tables, or views) +- Check applicable policies for any given resource + +## What is a Policy? + +A policy in Apache Polaris is a structured entity that defines rules governing actions on specified resources under +predefined conditions. Each policy contains: + +- **Name**: A unique identifier within a namespace +- **Type**: Determines the semantics and expected format of the policy content +- **Description**: Explains the purpose of the policy +- **Content**: Contains the actual rules defining the policy behavior +- **Version**: An automatically tracked revision number +- **Inheritable**: Whether the policy can be inherited by child resources, decided by its type + +### Policy Types + +Polaris supports several predefined system policy types (prefixed with `system.`): + +- **`system.data-compaction`**: Defines rules for data compaction operations + - Schema Definition: @https://polaris.apache.org/schemas/policies/system/data-compaction/2025-02-03.json + - Controls file compaction to optimize storage and query performance + +- **`system.metadata-compaction`**: Defines rules for metadata compaction operations + - Schema Definition: @https://polaris.apache.org/schemas/policies/system/metadata-compaction/2025-02-03.json + - Optimizes table metadata for improved performance + - Applicable resources: Iceberg table, namespace, catalog + +- **`system.orphan-file-removal`**: Defines rules for removing orphaned files + - Schema Definition: @https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json + - Identifies and safely removes files that are no longer referenced by the table metadata + - Applicable resources: Iceberg table, namespace, catalog + +- **`system.snapshot-expiry`**: Defines rules for snapshot expiration + - Schema Definition: @https://polaris.apache.org/schemas/policies/system/snapshot-expiry/2025-02-03.json + - Controls how long snapshots are retained before removal + - Applicable resources: Iceberg table, namespace, catalog + +- **Custom policy types**: Can be defined for specific organizational needs (WIP) + +- **FGAC (Fine-Grained Access Control) policies**: Row filtering, column masking, column hiding (WIP) + +### Policy Inheritance + +The entity hierarchy in Polaris is structured as follows: + +``` + Catalog + | + Namespace + | + +-----------+----------+ + | | | +Iceberg Iceberg Generic + Table View Table +``` + +Policies can be attached at any level, and inheritance flows from catalog down to namespace, then to tables and views. + +Policies can be inheritable or non-inheritable: + +- **Inheritable policies**: Apply to the target resource and all its applicable child resources +- **Non-inheritable policies**: Apply only to the specific target resource + +The inheritance follows an override mechanism: +1. Table-level policies override namespace and catalog policies +2. Namespace-level policies override parent namespace and catalog policies + +## Working with Policies + +### Creating a Policy + +To create a policy, you need to provide a name, type, and optionally a description and content: + +```json +POST /polaris/v1/{prefix}/namespaces/{namespace}/policies +{ + "name": "compaction-policy", + "type": "system.data-compaction", + "description": "Policy for optimizing table storage", + "content": "{\"version\": \"2025-02-03\", \"enable\": true, \"config\": {\"target_file_size_bytes\": 134217728}}" +} +``` + +The policy content is validated against a schema specific to its type. Here are a few policy content examples Review Comment: ```suggestion The policy content is validated against a schema specific to its type. Here are a few policy content examples: ``` ########## site/content/in-dev/unreleased/policy.md: ########## @@ -0,0 +1,207 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Policy +type: docs +weight: 425 +--- + +The Polaris Policy framework empowers organizations to centrally define, manage, and enforce fine-grained governance, lifecycle, and operational rules across all data resources in the catalog. + +With the policy API, you can: +- Create and manage policies +- Attach policies to specific resources (catalogs, namespaces, tables, or views) +- Check applicable policies for any given resource + +## What is a Policy? + +A policy in Apache Polaris is a structured entity that defines rules governing actions on specified resources under +predefined conditions. Each policy contains: + +- **Name**: A unique identifier within a namespace +- **Type**: Determines the semantics and expected format of the policy content +- **Description**: Explains the purpose of the policy +- **Content**: Contains the actual rules defining the policy behavior +- **Version**: An automatically tracked revision number +- **Inheritable**: Whether the policy can be inherited by child resources, decided by its type + +### Policy Types + +Polaris supports several predefined system policy types (prefixed with `system.`): + +- **`system.data-compaction`**: Defines rules for data compaction operations + - Schema Definition: @https://polaris.apache.org/schemas/policies/system/data-compaction/2025-02-03.json Review Comment: question: what does the "@" do for links in this framework? ########## site/content/in-dev/unreleased/policy.md: ########## @@ -0,0 +1,207 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: Policy +type: docs +weight: 425 +--- + +The Polaris Policy framework empowers organizations to centrally define, manage, and enforce fine-grained governance, lifecycle, and operational rules across all data resources in the catalog. + +With the policy API, you can: +- Create and manage policies +- Attach policies to specific resources (catalogs, namespaces, tables, or views) +- Check applicable policies for any given resource + +## What is a Policy? + +A policy in Apache Polaris is a structured entity that defines rules governing actions on specified resources under +predefined conditions. Each policy contains: + +- **Name**: A unique identifier within a namespace +- **Type**: Determines the semantics and expected format of the policy content +- **Description**: Explains the purpose of the policy +- **Content**: Contains the actual rules defining the policy behavior +- **Version**: An automatically tracked revision number +- **Inheritable**: Whether the policy can be inherited by child resources, decided by its type + +### Policy Types + +Polaris supports several predefined system policy types (prefixed with `system.`): + +- **`system.data-compaction`**: Defines rules for data compaction operations + - Schema Definition: @https://polaris.apache.org/schemas/policies/system/data-compaction/2025-02-03.json + - Controls file compaction to optimize storage and query performance + +- **`system.metadata-compaction`**: Defines rules for metadata compaction operations + - Schema Definition: @https://polaris.apache.org/schemas/policies/system/metadata-compaction/2025-02-03.json + - Optimizes table metadata for improved performance + - Applicable resources: Iceberg table, namespace, catalog + +- **`system.orphan-file-removal`**: Defines rules for removing orphaned files + - Schema Definition: @https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json + - Identifies and safely removes files that are no longer referenced by the table metadata + - Applicable resources: Iceberg table, namespace, catalog + +- **`system.snapshot-expiry`**: Defines rules for snapshot expiration + - Schema Definition: @https://polaris.apache.org/schemas/policies/system/snapshot-expiry/2025-02-03.json + - Controls how long snapshots are retained before removal + - Applicable resources: Iceberg table, namespace, catalog + +- **Custom policy types**: Can be defined for specific organizational needs (WIP) + +- **FGAC (Fine-Grained Access Control) policies**: Row filtering, column masking, column hiding (WIP) + +### Policy Inheritance + +The entity hierarchy in Polaris is structured as follows: + +``` + Catalog + | + Namespace + | + +-----------+----------+ + | | | +Iceberg Iceberg Generic + Table View Table +``` + +Policies can be attached at any level, and inheritance flows from catalog down to namespace, then to tables and views. + +Policies can be inheritable or non-inheritable: + +- **Inheritable policies**: Apply to the target resource and all its applicable child resources +- **Non-inheritable policies**: Apply only to the specific target resource + +The inheritance follows an override mechanism: +1. Table-level policies override namespace and catalog policies +2. Namespace-level policies override parent namespace and catalog policies + +## Working with Policies + +### Creating a Policy + +To create a policy, you need to provide a name, type, and optionally a description and content: + +```json +POST /polaris/v1/{prefix}/namespaces/{namespace}/policies +{ + "name": "compaction-policy", + "type": "system.data-compaction", + "description": "Policy for optimizing table storage", + "content": "{\"version\": \"2025-02-03\", \"enable\": true, \"config\": {\"target_file_size_bytes\": 134217728}}" +} +``` + +The policy content is validated against a schema specific to its type. Here are a few policy content examples +- Data Compaction Policy +```json +{ + "version": "2025-02-03", + "enable": true, + "config": { + "target_file_size_bytes": 134217728, + "compaction_strategy": "bin-pack", + "max-concurrent-file-group-rewrites": 5 + } +} +``` +- Orphan File Removal Policy +```json +{ + "version": "2025-02-03", + "enable": true, + "max_orphan_file_age_in_days": 30, + "locations": ["s3://my-bucket/my-table-location"], + "config": { + "prefix_mismatch_mode": "ignore" + } +} +``` + +### Attaching Policies to Resources + +Policies can be attached to different resource levels: + +1. **Catalog level**: Applies to the entire catalog +2. **Namespace level**: Applies to a specific namespace +3. **Table-like level**: Applies to individual tables or views + +Example of attaching a policy to a table: + +```json +PUT /polaris/v1/{prefix}/namespaces/{namespace}/policies/{policy-name}/mappings +{ + "target": { + "type": "table-like", + "path": ["NS1", "NS2", "test_table_1"] + } +} +``` + +For inheritable policies, only one policy of a given type can be attached to a resource. For non-inheritable policies, multiple policies of the same type can be attached. Review Comment: [not requesting change]: is there a document about why this is the case? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
