Re: [PR] add python version support range to pyproject.toml [iceberg-rust]

2025-01-20 Thread via GitHub
kevinjqliu commented on PR #903: URL: https://github.com/apache/iceberg-rust/pull/903#issuecomment-2603827979 thank you! @trim21 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] refactor: refine writer interface to support directory hierarchy [iceberg-rust]

2025-01-20 Thread via GitHub
ZENOTME commented on PR #893: URL: https://github.com/apache/iceberg-rust/pull/893#issuecomment-2603824410 > Thanks, @ZENOTME for the PR. I'm looking into it. Does the current approach handle multiple partition columns so that we can write into a multi-level directory structure? Or is it cu

Re: [PR] API: Add `UnknownType` [iceberg]

2025-01-20 Thread via GitHub
nastra commented on code in PR #12012: URL: https://github.com/apache/iceberg/pull/12012#discussion_r1923164162 ## api/src/test/java/org/apache/iceberg/types/TestSerializableTypes.java: ## @@ -136,6 +136,15 @@ public void testVariant() throws Exception { .isEqualTo(vari

Re: [PR] add python version support range to pyproject.toml [iceberg-rust]

2025-01-20 Thread via GitHub
trim21 commented on code in PR #903: URL: https://github.com/apache/iceberg-rust/pull/903#discussion_r1923135894 ## bindings/python/pyproject.toml: ## @@ -32,6 +33,7 @@ classifiers = [ "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "

Re: [PR] Spark 3.5: Refactor delete logic in batch reading [iceberg]

2025-01-20 Thread via GitHub
aokolnychyi commented on code in PR #11933: URL: https://github.com/apache/iceberg/pull/11933#discussion_r1923090697 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/ColumnarBatchUtil.java: ## @@ -0,0 +1,165 @@ +/* + * Licensed to the Apache Software Fo

Re: [PR] refactor: refine writer interface to support directory hierarchy [iceberg-rust]

2025-01-20 Thread via GitHub
hackintoshrao commented on PR #893: URL: https://github.com/apache/iceberg-rust/pull/893#issuecomment-2603752319 Thanks @ZENOTME for the PR< I'm looking into it. : Does the current approach handle multiple partition columns so that we can write into a multi-level directory structure? Or is

Re: [PR] feat(io): Add Create method for creating file writers [iceberg-go]

2025-01-20 Thread via GitHub
kevinjqliu commented on code in PR #262: URL: https://github.com/apache/iceberg-go/pull/262#discussion_r1923118119 ## io/local.go: ## @@ -30,6 +31,18 @@ func (LocalFS) Open(name string) (File, error) { return os.Open(strings.TrimPrefix(name, "file://")) } +func (Local

Re: [PR] WIP: Add headers for type/field/schema [iceberg-cpp]

2025-01-20 Thread via GitHub
lidavidm commented on PR #31: URL: https://github.com/apache/iceberg-cpp/pull/31#issuecomment-2603698520 I've implemented the Type subclasses. I've also added a trait for ToString/std::format formatting (might as well use the new C++ features if we're requiring C++20). Although it appears c

Re: [PR] add libavrocpp_s [iceberg-cpp]

2025-01-20 Thread via GitHub
zhjwpku commented on code in PR #34: URL: https://github.com/apache/iceberg-cpp/pull/34#discussion_r1923082582 ## src/iceberg/CMakeLists.txt: ## @@ -17,11 +17,16 @@ set(ICEBERG_SOURCES demo_table.cc) +set(ICEBERG_AVRO_STATIC_BUILD_LIBS) +list(APPEND ICEBERG_AVRO_STATIC_BUIL

Re: [PR] add libavrocpp_s [iceberg-cpp]

2025-01-20 Thread via GitHub
zhjwpku commented on code in PR #34: URL: https://github.com/apache/iceberg-cpp/pull/34#discussion_r1923078620 ## .github/workflows/test.yml: ## @@ -80,6 +80,10 @@ jobs: uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 with:

Re: [PR] add libavrocpp_s [iceberg-cpp]

2025-01-20 Thread via GitHub
zhjwpku commented on code in PR #34: URL: https://github.com/apache/iceberg-cpp/pull/34#discussion_r1923076221 ## .github/workflows/test.yml: ## @@ -80,6 +80,10 @@ jobs: uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 with:

Re: [PR] feat: support scan nested type(struct, map, list) [iceberg-rust]

2025-01-20 Thread via GitHub
ZENOTME commented on code in PR #882: URL: https://github.com/apache/iceberg-rust/pull/882#discussion_r1923063609 ## crates/iceberg/src/spec/datatypes.rs: ## @@ -226,8 +228,10 @@ pub enum PrimitiveType { /// Timestamp in microsecond precision, with timezone Timestamptz

Re: [PR] fix: fix timesmtap_ns serde name [iceberg-rust]

2025-01-20 Thread via GitHub
Xuanwo merged PR #905: URL: https://github.com/apache/iceberg-rust/pull/905 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] feat: support scan nested type(struct, map, list) [iceberg-rust]

2025-01-20 Thread via GitHub
Xuanwo commented on code in PR #882: URL: https://github.com/apache/iceberg-rust/pull/882#discussion_r1923062532 ## crates/iceberg/src/spec/datatypes.rs: ## @@ -226,8 +228,10 @@ pub enum PrimitiveType { /// Timestamp in microsecond precision, with timezone Timestamptz,

Re: [PR] Spark 3.5: Procedure to rewrite table path [iceberg]

2025-01-20 Thread via GitHub
dramaticlly commented on code in PR #11931: URL: https://github.com/apache/iceberg/pull/11931#discussion_r1923052475 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteTablePathProcedure.java: ## @@ -0,0 +1,174 @@ +/* + * Licensed to the

Re: [PR] feat: support scan nested type(struct, map, list) [iceberg-rust]

2025-01-20 Thread via GitHub
ZENOTME commented on code in PR #882: URL: https://github.com/apache/iceberg-rust/pull/882#discussion_r1923051061 ## crates/iceberg/src/arrow/schema.rs: ## @@ -43,7 +43,9 @@ use crate::spec::{ use crate::{Error, ErrorKind}; /// When iceberg map type convert to Arrow map type

Re: [PR] feat: support scan nested type(struct, map, list) [iceberg-rust]

2025-01-20 Thread via GitHub
ZENOTME commented on code in PR #882: URL: https://github.com/apache/iceberg-rust/pull/882#discussion_r1923050271 ## crates/iceberg/src/spec/datatypes.rs: ## @@ -226,8 +228,10 @@ pub enum PrimitiveType { /// Timestamp in microsecond precision, with timezone Timestamptz

Re: [PR] fix: fix timesmtap_ns serde name [iceberg-rust]

2025-01-20 Thread via GitHub
ZENOTME commented on PR #905: URL: https://github.com/apache/iceberg-rust/pull/905#issuecomment-2603596972 cc @liurenjie1024 @sdd @Xuanwo @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [I] How to apply partition/bloom filter to old data? Does rewrite_data_files/rewrite_manifests procedure work? [iceberg]

2025-01-20 Thread via GitHub
LoseYSelf commented on issue #11878: URL: https://github.com/apache/iceberg/issues/11878#issuecomment-2603586070 may be you can insert overwrite with the old data. @madeirak -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] WIP: Add headers for type/field/schema [iceberg-cpp]

2025-01-20 Thread via GitHub
lidavidm commented on code in PR #31: URL: https://github.com/apache/iceberg-cpp/pull/31#discussion_r1923025444 ## src/iceberg/type.h: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOT

Re: [I] Error Facing for Local Dev Setup [iceberg-python]

2025-01-20 Thread via GitHub
omkenge commented on issue #1541: URL: https://github.com/apache/iceberg-python/issues/1541#issuecomment-2603520714 @kevinjqliu Thank You for the solution -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [I] Error Facing for Local Dev Setup [iceberg-python]

2025-01-20 Thread via GitHub
omkenge closed issue #1541: Error Facing for Local Dev Setup URL: https://github.com/apache/iceberg-python/issues/1541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] feat: support scan nested type(struct, map, list) [iceberg-rust]

2025-01-20 Thread via GitHub
liurenjie1024 commented on code in PR #882: URL: https://github.com/apache/iceberg-rust/pull/882#discussion_r1922964452 ## crates/iceberg/src/arrow/schema.rs: ## @@ -43,7 +43,9 @@ use crate::spec::{ use crate::{Error, ErrorKind}; /// When iceberg map type convert to Arrow ma

Re: [PR] feat: support scan nested type(struct, map, list) [iceberg-rust]

2025-01-20 Thread via GitHub
liurenjie1024 commented on code in PR #882: URL: https://github.com/apache/iceberg-rust/pull/882#discussion_r1922964129 ## crates/iceberg/src/spec/datatypes.rs: ## @@ -226,8 +228,10 @@ pub enum PrimitiveType { /// Timestamp in microsecond precision, with timezone Times

Re: [PR] feat: support scan nested type(struct, map, list) [iceberg-rust]

2025-01-20 Thread via GitHub
liurenjie1024 commented on code in PR #882: URL: https://github.com/apache/iceberg-rust/pull/882#discussion_r1922963798 ## crates/iceberg/src/scan.rs: ## @@ -248,29 +248,6 @@ impl<'a> TableScanBuilder<'a> { ) })?; -let field = schema -

Re: [PR] Docs: Update Footer Copyright Year [iceberg]

2025-01-20 Thread via GitHub
amogh-jahagirdar merged PR #12011: URL: https://github.com/apache/iceberg/pull/12011 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Spark 3.3: Backport support for default values [iceberg]

2025-01-20 Thread via GitHub
manuzhang commented on PR #11988: URL: https://github.com/apache/iceberg/pull/11988#issuecomment-2603415279 @amogh-jahagirdar @rdblue I agree with your rational, but I'm confused about the criteria here. Shall we back-port other features from 3.4 / 3.5 since they are also nice and not diffi

Re: [PR] Update Footer Copyright Year [iceberg]

2025-01-20 Thread via GitHub
amogh-jahagirdar commented on PR #12011: URL: https://github.com/apache/iceberg/pull/12011#issuecomment-2603410067 Thanks @omkenge , this reminded me to address the NOTICE files :) https://github.com/apache/iceberg/pull/12013! -- This is an automated message from the Apache Git Service.

Re: [PR] feat: support datetime objects in literal instantiation [iceberg-python]

2025-01-20 Thread via GitHub
jayceslesar commented on code in PR #1542: URL: https://github.com/apache/iceberg-python/pull/1542#discussion_r1922933602 ## tests/integration/test_reads.py: ## @@ -950,3 +951,30 @@ def test_read_from_s3_and_local_fs(catalog: Catalog, tmp_path: PosixPath) -> Non result_t

[PR] Update notice files to reference 2025 [iceberg]

2025-01-20 Thread via GitHub
amogh-jahagirdar opened a new pull request, #12013: URL: https://github.com/apache/iceberg/pull/12013 Update notice files to reference 2025 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] feat: support datetime objects in literal instantiation [iceberg-python]

2025-01-20 Thread via GitHub
jayceslesar commented on code in PR #1542: URL: https://github.com/apache/iceberg-python/pull/1542#discussion_r1922933152 ## tests/integration/test_reads.py: ## @@ -950,3 +951,11 @@ def test_read_from_s3_and_local_fs(catalog: Catalog, tmp_path: PosixPath) -> Non result_t

Re: [PR] [WIP][Core] Restrict adding column of StructType with Empty Fields [iceberg]

2025-01-20 Thread via GitHub
github-actions[bot] commented on PR #11755: URL: https://github.com/apache/iceberg/pull/11755#issuecomment-2603393133 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [I] Allow static credentials for all AWS clients, not only for S3 [iceberg]

2025-01-20 Thread via GitHub
github-actions[bot] commented on issue #10614: URL: https://github.com/apache/iceberg/issues/10614#issuecomment-2603392971 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [PR] [WIP][Core] Restrict adding column of StructType with Empty Fields [iceberg]

2025-01-20 Thread via GitHub
github-actions[bot] closed pull request #11755: [WIP][Core] Restrict adding column of StructType with Empty Fields URL: https://github.com/apache/iceberg/pull/11755 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

2025-01-20 Thread via GitHub
github-actions[bot] commented on PR #11365: URL: https://github.com/apache/iceberg/pull/11365#issuecomment-2603393108 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If

Re: [PR] [Views] Update view spec with table identifier requirements [iceberg]

2025-01-20 Thread via GitHub
github-actions[bot] closed pull request #11365: [Views] Update view spec with table identifier requirements URL: https://github.com/apache/iceberg/pull/11365 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [I] `add_files` procedure allows importing NULL on NOT NULL columns [iceberg]

2025-01-20 Thread via GitHub
github-actions[bot] commented on issue #10742: URL: https://github.com/apache/iceberg/issues/10742#issuecomment-2603393027 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Will there be any problems migrating a hive table with 3 million partitions to Iceberg [iceberg]

2025-01-20 Thread via GitHub
github-actions[bot] commented on issue #10768: URL: https://github.com/apache/iceberg/issues/10768#issuecomment-2603393076 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Support for Default Values [iceberg]

2025-01-20 Thread via GitHub
github-actions[bot] commented on issue #10761: URL: https://github.com/apache/iceberg/issues/10761#issuecomment-2603393061 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occur

Re: [I] Spark rewrite Files Action OOM [iceberg]

2025-01-20 Thread via GitHub
github-actions[bot] closed issue #10054: Spark rewrite Files Action OOM URL: https://github.com/apache/iceberg/issues/10054 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] Spark rewrite Files Action OOM [iceberg]

2025-01-20 Thread via GitHub
github-actions[bot] commented on issue #10054: URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2603392960 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [PR] WIP: Add headers for type/field/schema [iceberg-cpp]

2025-01-20 Thread via GitHub
lidavidm commented on code in PR #31: URL: https://github.com/apache/iceberg-cpp/pull/31#discussion_r1922923718 ## src/iceberg/schema.h: ## @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NO

[I] There is a bug with the new version of boto used by s3fs that prevents writes to non-AWS S3 buckets [iceberg-python]

2025-01-20 Thread via GitHub
ryanovas opened a new issue, #1546: URL: https://github.com/apache/iceberg-python/issues/1546 ### Apache Iceberg version 0.8.1 (latest release) ### Please describe the bug 🐞 Here is a link to the relevant boto issue: https://github.com/boto/boto3/issues/4398 Attem

[PR] Build: Bump moto from 5.0.26 to 5.0.27 [iceberg-python]

2025-01-20 Thread via GitHub
dependabot[bot] opened a new pull request, #1545: URL: https://github.com/apache/iceberg-python/pull/1545 Bumps [moto](https://github.com/getmoto/moto) from 5.0.26 to 5.0.27. Changelog Sourced from https://github.com/getmoto/moto/blob/master/CHANGELOG.md";>moto's changelog.

[PR] Build: Bump pre-commit from 4.0.1 to 4.1.0 [iceberg-python]

2025-01-20 Thread via GitHub
dependabot[bot] opened a new pull request, #1544: URL: https://github.com/apache/iceberg-python/pull/1544 Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 4.0.1 to 4.1.0. Release notes Sourced from https://github.com/pre-commit/pre-commit/releases";>pre-commit's r

[PR] Build: Bump mkdocs-material from 9.5.49 to 9.5.50 [iceberg-python]

2025-01-20 Thread via GitHub
dependabot[bot] opened a new pull request, #1543: URL: https://github.com/apache/iceberg-python/pull/1543 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.49 to 9.5.50. Release notes Sourced from https://github.com/squidfunk/mkdocs-material/releases";>

Re: [PR] feat: support datetime objects in literal instantiation [iceberg-python]

2025-01-20 Thread via GitHub
kevinjqliu commented on code in PR #1542: URL: https://github.com/apache/iceberg-python/pull/1542#discussion_r1922894009 ## tests/integration/test_reads.py: ## @@ -950,3 +951,11 @@ def test_read_from_s3_and_local_fs(catalog: Catalog, tmp_path: PosixPath) -> Non result_ta

Re: [PR] feat: support datetime objects in literal instantiation [iceberg-python]

2025-01-20 Thread via GitHub
jayceslesar commented on PR #1542: URL: https://github.com/apache/iceberg-python/pull/1542#issuecomment-2603317427 > Thanks for the PR! Lets add a test for the example in the description with `row_filter` Added, let me know if we want anything more explicit there -- This is an auto

Re: [PR] Core, Spark: Include content offset/size in PositionDeletesTable [iceberg]

2025-01-20 Thread via GitHub
amogh-jahagirdar commented on code in PR #11808: URL: https://github.com/apache/iceberg/pull/11808#discussion_r1890997488 ## core/src/main/java/org/apache/iceberg/MetadataColumns.java: ## @@ -92,6 +92,20 @@ private MetadataColumns() {} Types.LongType.get(),

Re: [PR] feat: support datetime objects in literal instantiation [iceberg-python]

2025-01-20 Thread via GitHub
kevinjqliu commented on PR #1542: URL: https://github.com/apache/iceberg-python/pull/1542#issuecomment-2603291726 Thanks for the PR! Lets add a test for the example in the description with `row_filter` -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Core, Spark: Scan only live entries in RewriteTablePathUtil [iceberg]

2025-01-20 Thread via GitHub
szehon-ho commented on PR #12006: URL: https://github.com/apache/iceberg/pull/12006#issuecomment-2603148129 Yes, thanks for fixing the issue (found by our internal usage). I wonder, because the deleted entry may be important for CDC (to mark that this file at some point existed), is a

Re: [PR] Implement column projection [iceberg-python]

2025-01-20 Thread via GitHub
kevinjqliu commented on code in PR #1443: URL: https://github.com/apache/iceberg-python/pull/1443#discussion_r1922764225 ## pyiceberg/io/pyarrow.py: ## @@ -1237,16 +1265,26 @@ def _task_to_record_batches( # When V3 support is introduced, we will update `downcast_ns_tim

Re: [PR] feat(catalog): Have Load use "type" property and "name" for config [iceberg-go]

2025-01-20 Thread via GitHub
zeroshade merged PR #260: URL: https://github.com/apache/iceberg-go/pull/260 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [I] Error Facing for Local Dev Setup [iceberg-python]

2025-01-20 Thread via GitHub
kevinjqliu commented on issue #1541: URL: https://github.com/apache/iceberg-python/issues/1541#issuecomment-2603123056 i think `catalog=load_catalog("default")` is not loading the `.pyiceberg.yaml` configs, since the name `default` is not in the yaml file. -- This is an automate

Re: [PR] Arrow: Avoid buffer-overflow by avoid doing a sort [iceberg-python]

2025-01-20 Thread via GitHub
kevinjqliu commented on code in PR #1539: URL: https://github.com/apache/iceberg-python/pull/1539#discussion_r1922786157 ## pyiceberg/partitioning.py: ## @@ -413,7 +414,9 @@ def partition_record_value(partition_field: PartitionField, value: Any, schema: the final partition

Re: [I] `datetime` objects in `row_filter` expressions are not casted and raise an error [iceberg-python]

2025-01-20 Thread via GitHub
kevinjqliu commented on issue #1456: URL: https://github.com/apache/iceberg-python/issues/1456#issuecomment-2603098530 I think `TimestampLiteral(datetime_to_micros(value))` is fine. We use `TimestampLiteral(timestamp_to_micros(self.value))` to convert string to timestamp https:/

[I] Error Facing for Local Dev Setup [iceberg-python]

2025-01-20 Thread via GitHub
omkenge opened a new issue, #1541: URL: https://github.com/apache/iceberg-python/issues/1541 ### Question **.pyiceberg.yaml** default-catalog: local catalog: local: uri: http://127.0.0.1:8181 s3.endpoint: http://127.0.0.1:9000 py-io-impl: p

Re: [PR] Arrow: Avoid buffer-overflow by avoid doing a sort [iceberg-python]

2025-01-20 Thread via GitHub
Fokko commented on code in PR #1539: URL: https://github.com/apache/iceberg-python/pull/1539#discussion_r1922771682 ## pyiceberg/io/pyarrow.py: ## Review Comment: Good one, updated and simplified! -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] Arrow: Avoid buffer-overflow by avoid doing a sort [iceberg-python]

2025-01-20 Thread via GitHub
Fokko commented on code in PR #1539: URL: https://github.com/apache/iceberg-python/pull/1539#discussion_r1922769699 ## pyiceberg/partitioning.py: ## @@ -425,8 +426,13 @@ def _to_partition_representation(type: IcebergType, value: Any) -> Any: @_to_partition_representation.reg

Re: [PR] Arrow: Avoid buffer-overflow by avoid doing a sort [iceberg-python]

2025-01-20 Thread via GitHub
Fokko commented on code in PR #1539: URL: https://github.com/apache/iceberg-python/pull/1539#discussion_r1922765663 ## pyiceberg/io/pyarrow.py: ## @@ -2594,42 +2566,46 @@ def _determine_partitions(spec: PartitionSpec, schema: Schema, arrow_table: pa.T We then retrieve the

Re: [PR] Arrow: Avoid buffer-overflow by avoid doing a sort [iceberg-python]

2025-01-20 Thread via GitHub
Fokko commented on code in PR #1539: URL: https://github.com/apache/iceberg-python/pull/1539#discussion_r1922764937 ## pyiceberg/partitioning.py: ## @@ -425,8 +426,13 @@ def _to_partition_representation(type: IcebergType, value: Any) -> Any: @_to_partition_representation.reg

Re: [I] `datetime` objects in `row_filter` expressions are not casted and raise an error [iceberg-python]

2025-01-20 Thread via GitHub
jayceslesar commented on issue #1456: URL: https://github.com/apache/iceberg-python/issues/1456#issuecomment-2603070402 > [@jayceslesar](https://github.com/jayceslesar) i think that makes sense. Are you interested to contribute this? Happy to contribute, was just wondering if there w

Re: [PR] Docs: Location Provider Documentation [iceberg-python]

2025-01-20 Thread via GitHub
kevinjqliu commented on code in PR #1537: URL: https://github.com/apache/iceberg-python/pull/1537#discussion_r1922753001 ## mkdocs/docs/configuration.md: ## @@ -195,6 +198,86 @@ PyIceberg uses [S3FileSystem](https://arrow.apache.org/docs/python/generated/pya +## Location

Re: [PR] Support WASB scheme in ADLSFileIO [iceberg]

2025-01-20 Thread via GitHub
amogh-jahagirdar merged PR #11830: URL: https://github.com/apache/iceberg/pull/11830 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@

Re: [PR] Support WASB scheme in ADLSFileIO [iceberg]

2025-01-20 Thread via GitHub
amogh-jahagirdar commented on PR #11830: URL: https://github.com/apache/iceberg/pull/11830#issuecomment-2603021202 Thanks for the change @mrcnc , and thank you for reviewing @jbonofre! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Arrow: Avoid buffer-overflow by avoid doing a sort [iceberg-python]

2025-01-20 Thread via GitHub
kevinjqliu commented on code in PR #1539: URL: https://github.com/apache/iceberg-python/pull/1539#discussion_r1922707073 ## tests/benchmark/test_benchmark.py: ## @@ -0,0 +1,72 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreeme

Re: [PR] docs: Add docstrings for Identifier, Properties, RecursiveDict [iceberg-python]

2025-01-20 Thread via GitHub
rodrigc commented on PR #1530: URL: https://github.com/apache/iceberg-python/pull/1530#issuecomment-2603011752 @Fokko is this PR OK land? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] WIP: Deletion vectors [iceberg-python]

2025-01-20 Thread via GitHub
Fokko commented on PR #1516: URL: https://github.com/apache/iceberg-python/pull/1516#issuecomment-2602994581 @kevinjqliu That's correct, I've split that out here: https://github.com/apache/iceberg-python/issues/1540 -- This is an automated message from the Apache Git Service. To respond t

[I] Support reading V3 tables [iceberg-python]

2025-01-20 Thread via GitHub
Fokko opened a new issue, #1540: URL: https://github.com/apache/iceberg-python/issues/1540 ### Feature Request / Improvement Currently, we raise an exception when reading V3 tables. As a first step, we should be able to read V3 tables. An overview can be found here: https://ic

Re: [PR] Support WASB scheme in ADLSFileIO [iceberg]

2025-01-20 Thread via GitHub
jbonofre commented on code in PR #11830: URL: https://github.com/apache/iceberg/pull/11830#discussion_r1922720993 ## core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java: ## @@ -62,7 +62,9 @@ public class ResolvingFileIO implements HadoopConfigurable, DelegateFileIO {

Re: [I] Support UnknownType for V3 Schema [iceberg]

2025-01-20 Thread via GitHub
Fokko commented on issue #11732: URL: https://github.com/apache/iceberg/issues/11732#issuecomment-2602952143 Allow me to pick this one up @HonahX -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] feat: support merge append action [iceberg-rust]

2025-01-20 Thread via GitHub
kevinjqliu commented on code in PR #902: URL: https://github.com/apache/iceberg-rust/pull/902#discussion_r1922681882 ## crates/iceberg/src/spec/manifest.rs: ## @@ -1182,6 +1182,12 @@ impl ManifestEntry { pub fn data_file(&self) -> &DataFile { &self.data_file }

Re: [I] `datetime` objects in `row_filter` expressions are not casted and raise an error [iceberg-python]

2025-01-20 Thread via GitHub
kevinjqliu commented on issue #1456: URL: https://github.com/apache/iceberg-python/issues/1456#issuecomment-2602932859 @jayceslesar i think that makes sense. Are you interested to contribute this? -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] add python version support range to pyproject.toml [iceberg-rust]

2025-01-20 Thread via GitHub
kevinjqliu commented on code in PR #903: URL: https://github.com/apache/iceberg-rust/pull/903#discussion_r1922693234 ## bindings/python/pyproject.toml: ## @@ -32,6 +33,7 @@ classifiers = [ "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11",

Re: [PR] add python version support range to pyproject.toml [iceberg-rust]

2025-01-20 Thread via GitHub
kevinjqliu commented on code in PR #903: URL: https://github.com/apache/iceberg-rust/pull/903#discussion_r1922692179 ## bindings/python/pyproject.toml: ## @@ -23,6 +23,7 @@ build-backend = "maturin" name = "pyiceberg_core" version = "0.4.0" readme = "project-description.md" +

Re: [PR] add python version support range to pyproject.toml [iceberg-rust]

2025-01-20 Thread via GitHub
trim21 commented on code in PR #903: URL: https://github.com/apache/iceberg-rust/pull/903#discussion_r1922681062 ## bindings/python/pyproject.toml: ## @@ -23,6 +23,7 @@ build-backend = "maturin" name = "pyiceberg_core" version = "0.4.0" readme = "project-description.md" +requ

Re: [PR] add python version support range to pyproject.toml [iceberg-rust]

2025-01-20 Thread via GitHub
trim21 commented on code in PR #903: URL: https://github.com/apache/iceberg-rust/pull/903#discussion_r1922682955 ## bindings/python/pyproject.toml: ## @@ -32,6 +33,7 @@ classifiers = [ "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "

Re: [PR] add python version support range to pyproject.toml [iceberg-rust]

2025-01-20 Thread via GitHub
trim21 commented on code in PR #903: URL: https://github.com/apache/iceberg-rust/pull/903#discussion_r1922681062 ## bindings/python/pyproject.toml: ## @@ -23,6 +23,7 @@ build-backend = "maturin" name = "pyiceberg_core" version = "0.4.0" readme = "project-description.md" +requ

Re: [PR] add python version support range to pyproject.toml [iceberg-rust]

2025-01-20 Thread via GitHub
kevinjqliu commented on code in PR #903: URL: https://github.com/apache/iceberg-rust/pull/903#discussion_r1922678898 ## bindings/python/pyproject.toml: ## @@ -32,6 +33,7 @@ classifiers = [ "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11",

Re: [PR] WIP: Deletion vectors [iceberg-python]

2025-01-20 Thread via GitHub
kevinjqliu commented on PR #1516: URL: https://github.com/apache/iceberg-python/pull/1516#issuecomment-2602894613 Looks like V3 is not supported on the pyiceberg side ``` === short test summary info FAILED tests/integration/t

Re: [PR] WIP: Add headers for type/field/schema [iceberg-cpp]

2025-01-20 Thread via GitHub
gaborkaszab commented on code in PR #31: URL: https://github.com/apache/iceberg-cpp/pull/31#discussion_r1922625409 ## src/iceberg/type.h: ## @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the

Re: [PR] WIP: Add headers for type/field/schema [iceberg-cpp]

2025-01-20 Thread via GitHub
gaborkaszab commented on PR #31: URL: https://github.com/apache/iceberg-cpp/pull/31#issuecomment-2602857029 > Just to make sure @gaborkaszab @wgtmac: are we ok with the Arrow-style type representation here? (Types are represented by a class hierarchy, erased behind smart pointers; nested ty

Re: [PR] add libavrocpp_s [iceberg-cpp]

2025-01-20 Thread via GitHub
wgtmac commented on code in PR #34: URL: https://github.com/apache/iceberg-cpp/pull/34#discussion_r1922579415 ## src/iceberg/CMakeLists.txt: ## @@ -17,11 +17,16 @@ set(ICEBERG_SOURCES demo_table.cc) +set(ICEBERG_AVRO_STATIC_BUILD_LIBS) +list(APPEND ICEBERG_AVRO_STATIC_BUILD

Re: [PR] add libavrocpp_s [iceberg-cpp]

2025-01-20 Thread via GitHub
wgtmac commented on code in PR #34: URL: https://github.com/apache/iceberg-cpp/pull/34#discussion_r1922577230 ## src/iceberg/CMakeLists.txt: ## @@ -17,11 +17,16 @@ set(ICEBERG_SOURCES demo_table.cc) +set(ICEBERG_AVRO_STATIC_BUILD_LIBS) +list(APPEND ICEBERG_AVRO_STATIC_BUILD

Re: [PR] add libavrocpp_s [iceberg-cpp]

2025-01-20 Thread via GitHub
wgtmac commented on code in PR #34: URL: https://github.com/apache/iceberg-cpp/pull/34#discussion_r1922569711 ## .github/workflows/test.yml: ## @@ -80,6 +80,10 @@ jobs: uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 with: f

Re: [PR] add libavrocpp_s [iceberg-cpp]

2025-01-20 Thread via GitHub
wgtmac commented on code in PR #34: URL: https://github.com/apache/iceberg-cpp/pull/34#discussion_r1922562686 ## .github/workflows/test.yml: ## @@ -80,6 +80,10 @@ jobs: uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 with: f

Re: [PR] OpenAPI: Deprecate snapshot-id of SetStatisticsUpdate [iceberg]

2025-01-20 Thread via GitHub
nastra commented on PR #12010: URL: https://github.com/apache/iceberg/pull/12010#issuecomment-2602652236 > @nastra ready for another round. Do we need a Vote on this in the Mailing List? Yes I think we should have a quick formal vote so that people are aware of this change -- This

Re: [PR] Add ResidualVisitor to compute residuals [iceberg-python]

2025-01-20 Thread via GitHub
Fokko commented on code in PR #1388: URL: https://github.com/apache/iceberg-python/pull/1388#discussion_r1922535895 ## pyiceberg/table/__init__.py: ## @@ -1596,6 +1630,43 @@ def to_ray(self) -> ray.data.dataset.Dataset: return ray.data.from_arrow(self.to_arrow()) +

Re: [PR] WIP: Deletion vectors [iceberg-python]

2025-01-20 Thread via GitHub
Fokko commented on code in PR #1516: URL: https://github.com/apache/iceberg-python/pull/1516#discussion_r1922530121 ## dev/provision.py: ## @@ -401,3 +401,43 @@ ) spark.sql(f"ALTER TABLE {catalog_name}.default.test_empty_scan_ordered_str WRITE ORDERED BY id") spa

Re: [PR] WIP: Add headers for type/field/schema [iceberg-cpp]

2025-01-20 Thread via GitHub
mapleFU commented on PR #31: URL: https://github.com/apache/iceberg-cpp/pull/31#issuecomment-2602610638 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Spark 3.5: Procedure to rewrite table path [iceberg]

2025-01-20 Thread via GitHub
nastra commented on code in PR #11931: URL: https://github.com/apache/iceberg/pull/11931#discussion_r1922479965 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteTablePathProcedure.java: ## @@ -0,0 +1,174 @@ +/* + * Licensed to the Apach

Re: [PR] OpenAPI: Deprecate snapshot-id of SetStatisticsUpdate [iceberg]

2025-01-20 Thread via GitHub
c-thiel commented on PR #12010: URL: https://github.com/apache/iceberg/pull/12010#issuecomment-2602554274 @nastra ready for another round -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] OpenAPI: Deprecate snapshot-id of SetStatisticsUpdate [iceberg]

2025-01-20 Thread via GitHub
c-thiel commented on code in PR #12010: URL: https://github.com/apache/iceberg/pull/12010#discussion_r1922478361 ## api/src/main/java/org/apache/iceberg/UpdateStatistics.java: ## @@ -27,9 +27,21 @@ public interface UpdateStatistics extends PendingUpdate> { * the snapshot if

Re: [PR] Spark 3.5: Procedure to rewrite table path [iceberg]

2025-01-20 Thread via GitHub
nastra commented on code in PR #11931: URL: https://github.com/apache/iceberg/pull/11931#discussion_r1922477983 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteTablePathProcedure.java: ## @@ -0,0 +1,174 @@ +/* + * Licensed to the Apach

Re: [PR] Core, API, Spec: Metadata Row Lineage [iceberg]

2025-01-20 Thread via GitHub
nastra commented on code in PR #11948: URL: https://github.com/apache/iceberg/pull/11948#discussion_r1922468617 ## core/src/test/java/org/apache/iceberg/TestRowLineageMetadata.java: ## @@ -0,0 +1,339 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or mo

Re: [PR] Spark 3.5: Procedure to rewrite table path [iceberg]

2025-01-20 Thread via GitHub
nastra commented on code in PR #11931: URL: https://github.com/apache/iceberg/pull/11931#discussion_r1922473129 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteTablePathProcedure.java: ## @@ -0,0 +1,174 @@ +/* + * Licensed to the Apach

Re: [PR] WIP: Add headers for type/field/schema [iceberg-cpp]

2025-01-20 Thread via GitHub
wgtmac commented on PR #31: URL: https://github.com/apache/iceberg-cpp/pull/31#issuecomment-2602518959 I vote +1 for the Arrow-style. I haven't checked the `cuDF` style in detail but it looks more like the C-style. The extra benefit of Arrow-style is that it might be simpler to implement th

Re: [PR] List files [iceberg-rust]

2025-01-20 Thread via GitHub
SergeiPatiakin closed pull request #904: List files URL: https://github.com/apache/iceberg-rust/pull/904 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] List files [iceberg-rust]

2025-01-20 Thread via GitHub
SergeiPatiakin commented on PR #904: URL: https://github.com/apache/iceberg-rust/pull/904#issuecomment-2602517336 Opened by mistake -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[PR] List files [iceberg-rust]

2025-01-20 Thread via GitHub
SergeiPatiakin opened a new pull request, #904: URL: https://github.com/apache/iceberg-rust/pull/904 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

  1   2   >