[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.40.0 to 1.40.1 [iceberg-go]

2023-10-08 Thread via GitHub
dependabot[bot] opened a new pull request, #13: URL: https://github.com/apache/iceberg-go/pull/13 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.40.0 to 1.40.1. Commits https://github.com/aws/aws-sdk-go-v2/commit/e155bb72a2ec20ec61

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.21.0 to 1.21.1 [iceberg-go]

2023-10-08 Thread via GitHub
dependabot[bot] opened a new pull request, #14: URL: https://github.com/apache/iceberg-go/pull/14 Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.21.0 to 1.21.1. Commits https://github.com/aws/aws-sdk-go-v2/commit/e155bb72a2ec20ec61db50fc3d4568

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.13.40 to 1.13.42 [iceberg-go]

2023-10-08 Thread via GitHub
dependabot[bot] opened a new pull request, #15: URL: https://github.com/apache/iceberg-go/pull/15 Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.13.40 to 1.13.42. Commits https://github.com/aws/aws-sdk-go-v2/commit/e155bb72a2ec20e

[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.18.42 to 1.18.44 [iceberg-go]

2023-10-08 Thread via GitHub
dependabot[bot] opened a new pull request, #16: URL: https://github.com/apache/iceberg-go/pull/16 Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.18.42 to 1.18.44. Commits https://github.com/aws/aws-sdk-go-v2/commit/e155bb72a2ec20ec61db

[I] pos delete files not removed immediately after rewrite [iceberg]

2023-10-08 Thread via GitHub
zinking opened a new issue, #8743: URL: https://github.com/apache/iceberg/issues/8743 ### Apache Iceberg version 1.4.0 (latest release) ### Query engine Spark ### Please describe the bug 🐞 org.apache.iceberg.spark.actions.TestRewriteDataFilesAction#test

[I] Can not split files on s3 when use flink to read iceberg on s3 with small split-size [iceberg]

2023-10-08 Thread via GitHub
Zelin1203 opened a new issue, #8744: URL: https://github.com/apache/iceberg/issues/8744 ### Query engine Flink ### Question I set the split-size smaller than files on s3. Each snapshot has 16 added-files. When reading data, only 16 splits can be obtained at a time, w

[I] Implementation does not write `schema-id` into Manifest Avro headers [iceberg]

2023-10-08 Thread via GitHub
JFinis opened a new issue, #8745: URL: https://github.com/apache/iceberg/issues/8745 ### Apache Iceberg version 1.3.1 ### Query engine Spark ### Please describe the bug 🐞 The spec mandates the field `schema-id` to be written into the Avro header of Manifest

[I] Spec does not define which header fields to be present in ManifestLists [iceberg]

2023-10-08 Thread via GitHub
JFinis opened a new issue, #8746: URL: https://github.com/apache/iceberg/issues/8746 ### Apache Iceberg version 1.3.1 ### Query engine None ### Please describe the bug 🐞 The [Iceberg spec mandates which fields to be present in Manifest files](https://iceber

Re: [I] Can not split files on s3 when use flink to read iceberg on s3 with small split-size [iceberg]

2023-10-08 Thread via GitHub
RussellSpitzer commented on issue #8744: URL: https://github.com/apache/iceberg/issues/8744#issuecomment-1752029027 One important thing to remember is that Parquet files are not arbitrarily splittable. If there is only a single row group in a parquet file, it cannot be split. Are you sure y

Re: [I] Manifest List Writer Design [iceberg-rust]

2023-10-08 Thread via GitHub
ZENOTME commented on issue #72: URL: https://github.com/apache/iceberg-rust/issues/72#issuecomment-1752043696 > Should we support appending entries like the Java and Python implementations instead of a single write? >> The ManifestList is a simple wrapper of Vec, so I think providing a m

Re: [I] Can not split files on s3 when use flink to read iceberg on s3 with small split-size [iceberg]

2023-10-08 Thread via GitHub
Zelin1203 commented on issue #8744: URL: https://github.com/apache/iceberg/issues/8744#issuecomment-1752047235 > One important thing to remember is that Parquet files are not arbitrarily splittable. If there is only a single row group in a parquet file, it cannot be split. Are you sure your

Re: [I] Can not split files on s3 when use flink to read iceberg on s3 with small split-size [iceberg]

2023-10-08 Thread via GitHub
Zelin1203 closed issue #8744: Can not split files on s3 when use flink to read iceberg on s3 with small split-size URL: https://github.com/apache/iceberg/issues/8744 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Build: Bump fastavro from 1.8.3 to 1.8.4 in /python [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat closed pull request #8742: Build: Bump fastavro from 1.8.3 to 1.8.4 in /python URL: https://github.com/apache/iceberg/pull/8742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Build: Bump fastavro from 1.8.3 to 1.8.4 in /python [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat commented on PR #8742: URL: https://github.com/apache/iceberg/pull/8742#issuecomment-1752106689 python development is moved to https://github.com/apache/iceberg-python and python directory will be removed from https://github.com/apache/iceberg/pull/8695 -- This is an automat

Re: [PR] Build: Bump fastavro from 1.8.3 to 1.8.4 in /python [iceberg]

2023-10-08 Thread via GitHub
dependabot[bot] commented on PR #8742: URL: https://github.com/apache/iceberg/pull/8742#issuecomment-1752106700 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Build: Bump coverage from 7.3.1 to 7.3.2 in /python [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat commented on PR #8741: URL: https://github.com/apache/iceberg/pull/8741#issuecomment-1752106787 Python development is moved to https://github.com/apache/iceberg-python and Python directory will be removed from https://github.com/apache/iceberg/pull/8695 -- This is an automat

Re: [PR] Build: Bump coverage from 7.3.1 to 7.3.2 in /python [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat closed pull request #8741: Build: Bump coverage from 7.3.1 to 7.3.2 in /python URL: https://github.com/apache/iceberg/pull/8741 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Build: Bump coverage from 7.3.1 to 7.3.2 in /python [iceberg]

2023-10-08 Thread via GitHub
dependabot[bot] commented on PR #8741: URL: https://github.com/apache/iceberg/pull/8741#issuecomment-1752106810 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Build: Bump pydantic from 2.3.0 to 2.4.2 in /python [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat commented on PR #8740: URL: https://github.com/apache/iceberg/pull/8740#issuecomment-1752106911 Python development is moved to https://github.com/apache/iceberg-python and Python directory will be removed from https://github.com/apache/iceberg/pull/8695 -- This is an automat

Re: [PR] Build: Bump pydantic from 2.3.0 to 2.4.2 in /python [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat closed pull request #8740: Build: Bump pydantic from 2.3.0 to 2.4.2 in /python URL: https://github.com/apache/iceberg/pull/8740 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Build: Bump pydantic from 2.3.0 to 2.4.2 in /python [iceberg]

2023-10-08 Thread via GitHub
dependabot[bot] commented on PR #8740: URL: https://github.com/apache/iceberg/pull/8740#issuecomment-1752106928 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Build: Bump mkdocstrings-python from 1.7.0 to 1.7.2 in /python [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat commented on PR #8739: URL: https://github.com/apache/iceberg/pull/8739#issuecomment-1752106960 Python development is moved to https://github.com/apache/iceberg-python and Python directory will be removed from https://github.com/apache/iceberg/pull/8695 -- This is an automat

Re: [PR] Build: Bump mkdocstrings-python from 1.7.0 to 1.7.2 in /python [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat closed pull request #8739: Build: Bump mkdocstrings-python from 1.7.0 to 1.7.2 in /python URL: https://github.com/apache/iceberg/pull/8739 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Build: Bump mkdocstrings-python from 1.7.0 to 1.7.2 in /python [iceberg]

2023-10-08 Thread via GitHub
dependabot[bot] commented on PR #8739: URL: https://github.com/apache/iceberg/pull/8739#issuecomment-1752106973 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Build: Bump cython from 3.0.2 to 3.0.3 in /python [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat commented on PR #8738: URL: https://github.com/apache/iceberg/pull/8738#issuecomment-1752107013 Python development is moved to https://github.com/apache/iceberg-python and Python directory will be removed from https://github.com/apache/iceberg/pull/8695 -- This is an automat

Re: [PR] Build: Bump cython from 3.0.2 to 3.0.3 in /python [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat closed pull request #8738: Build: Bump cython from 3.0.2 to 3.0.3 in /python URL: https://github.com/apache/iceberg/pull/8738 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Build: Bump cython from 3.0.2 to 3.0.3 in /python [iceberg]

2023-10-08 Thread via GitHub
dependabot[bot] commented on PR #8738: URL: https://github.com/apache/iceberg/pull/8738#issuecomment-1752107022 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Build: Bump pypa/cibuildwheel from 2.16.0 to 2.16.2 [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat closed pull request #8733: Build: Bump pypa/cibuildwheel from 2.16.0 to 2.16.2 URL: https://github.com/apache/iceberg/pull/8733 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Build: Bump pypa/cibuildwheel from 2.16.0 to 2.16.2 [iceberg]

2023-10-08 Thread via GitHub
dependabot[bot] commented on PR #8733: URL: https://github.com/apache/iceberg/pull/8733#issuecomment-1752107143 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let

Re: [PR] Build: Bump pypa/cibuildwheel from 2.16.0 to 2.16.2 [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat commented on PR #8733: URL: https://github.com/apache/iceberg/pull/8733#issuecomment-1752107130 Python development is moved to https://github.com/apache/iceberg-python and Python directory will be removed from https://github.com/apache/iceberg/pull/8695 -- This is an automat

Re: [I] pos delete files not removed immediately after rewrite [iceberg]

2023-10-08 Thread via GitHub
RussellSpitzer commented on issue #8743: URL: https://github.com/apache/iceberg/issues/8743#issuecomment-1752113516 As expected, there is extensive discussion on this. The main issue being that only in the case of a "rewrite-all" are we sure we can remove delete files. There were some thoug

Re: [PR] Python: Update how to verify [iceberg]

2023-10-08 Thread via GitHub
Fokko closed pull request #8504: Python: Update how to verify URL: https://github.com/apache/iceberg/pull/8504 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] Spec: Clarify spec_id field in Data File [iceberg]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #8730: URL: https://github.com/apache/iceberg/pull/8730#discussion_r1349759341 ## format/spec.md: ## @@ -443,13 +443,13 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo | _optional_ | _optional_ | **`132 split_

Re: [PR] Spec: Clarify spec_id field in Data File [iceberg]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #8730: URL: https://github.com/apache/iceberg/pull/8730#discussion_r1349759341 ## format/spec.md: ## @@ -443,13 +443,13 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo | _optional_ | _optional_ | **`132 split_

Re: [I] Null support in Apache Flink [iceberg]

2023-10-08 Thread via GitHub
Ge commented on issue #8720: URL: https://github.com/apache/iceberg/issues/8720#issuecomment-1752127496 ``` Flink SQL> SELECT * FROM word_count WHERE cnt = NULL; +++--+ | op | word |

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349761999 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVi

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349762135 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVi

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349762539 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVi

Re: [PR] chore(deps): Update apache-avro requirement from 0.15 to 0.16 [iceberg-rust]

2023-10-08 Thread via GitHub
Fokko merged PR #71: URL: https://github.com/apache/iceberg-rust/pull/71 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349762706 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVi

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349762706 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVi

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349763567 ## pyiceberg/avro/writer.py: ## @@ -49,58 +51,69 @@ def __repr__(self) -> str: return f"{self.__class__.__name__}()" +@dataclass(frozen=True) class None

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349764107 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVi

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349764258 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVi

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349764721 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVi

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349764876 ## pyiceberg/manifest.py: ## @@ -99,101 +100,185 @@ def __repr__(self) -> str: return f"FileFormat.{self.name}" -DATA_FILE_TYPE_V1 = StructType( -Nes

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349765776 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVi

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349766027 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVi

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349766166 ## pyiceberg/avro/resolver.py: ## @@ -192,7 +195,26 @@ def visit_binary(self, binary_type: BinaryType) -> Writer: return BinaryWriter() -def resolve( +CO

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349766913 ## pyiceberg/manifest.py: ## @@ -603,10 +691,26 @@ def __exit__( def content(self) -> ManifestContent: ... +@property @abstractmethod -de

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349767535 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVi

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
Fokko commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349776424 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVis

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
Fokko commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349776456 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVis

Re: [PR] allow override env-variables in load_catalog [iceberg-python]

2023-10-08 Thread via GitHub
rdblue commented on PR #45: URL: https://github.com/apache/iceberg-python/pull/45#issuecomment-1752161169 The proposed behavior sounds reasonable to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
Fokko commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349776586 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVis

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
Fokko commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349776729 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVis

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
Fokko commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349776857 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVis

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
Fokko commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349777029 ## pyiceberg/avro/resolver.py: ## @@ -233,7 +255,107 @@ def skip(self, decoder: BinaryDecoder) -> None: pass -class SchemaResolver(PrimitiveWithPartnerVis

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
Fokko commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349777115 ## pyiceberg/avro/writer.py: ## @@ -49,58 +51,69 @@ def __repr__(self) -> str: return f"{self.__class__.__name__}()" +@dataclass(frozen=True) class NoneW

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
Fokko commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349777193 ## pyiceberg/manifest.py: ## @@ -99,101 +100,185 @@ def __repr__(self) -> str: return f"FileFormat.{self.name}" -DATA_FILE_TYPE_V1 = StructType( -Nest

[PR] Flink:backport PR to 1.16 #7360: Implement data statistics coordinator to aggregate data statistics from operator subtasks [iceberg]

2023-10-08 Thread via GitHub
yegangy0718 opened a new pull request, #8747: URL: https://github.com/apache/iceberg/pull/8747 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Construct a writer tree [iceberg-python]

2023-10-08 Thread via GitHub
Fokko commented on code in PR #40: URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1349777648 ## pyiceberg/manifest.py: ## @@ -603,10 +691,26 @@ def __exit__( def content(self) -> ManifestContent: ... +@property @abstractmethod -def

Re: [PR] allow override env-variables in load_catalog [iceberg-python]

2023-10-08 Thread via GitHub
Fokko commented on PR #45: URL: https://github.com/apache/iceberg-python/pull/45#issuecomment-1752165287 Thanks @bdilday for raising this, and I agree that the explicitly passed-in configuration should take preference. Can you rebase against the latest main branch? The CI should turn green

Re: [PR] Spec: Clarify missing fields when writing [iceberg]

2023-10-08 Thread via GitHub
Fokko commented on code in PR #8672: URL: https://github.com/apache/iceberg/pull/8672#discussion_r1349779338 ## format/spec.md: ## @@ -128,13 +128,13 @@ Tables do not require rename, except for tables that use atomic rename to implem Writer requirements -Some tables i

Re: [PR] Spec: Clarify missing fields when writing [iceberg]

2023-10-08 Thread via GitHub
Fokko commented on code in PR #8672: URL: https://github.com/apache/iceberg/pull/8672#discussion_r1349779366 ## format/spec.md: ## @@ -128,13 +128,13 @@ Tables do not require rename, except for tables that use atomic rename to implem Writer requirements -Some tables i

Re: [PR] Spec: Clarify missing fields when writing [iceberg]

2023-10-08 Thread via GitHub
Fokko commented on code in PR #8672: URL: https://github.com/apache/iceberg/pull/8672#discussion_r1349779513 ## format/spec.md: ## @@ -128,13 +128,13 @@ Tables do not require rename, except for tables that use atomic rename to implem Writer requirements -Some tables i

Re: [PR] Core: Mark `503: added_snapshot_id` as required [iceberg]

2023-10-08 Thread via GitHub
Fokko commented on code in PR #8673: URL: https://github.com/apache/iceberg/pull/8673#discussion_r1349779777 ## core/src/main/java/org/apache/iceberg/V2Metadata.java: ## @@ -39,7 +39,7 @@ private V2Metadata() {} ManifestFile.MANIFEST_CONTENT.asRequired(), M

[PR] Update spark-getting-started.md [iceberg]

2023-10-08 Thread via GitHub
Priyansh121096 opened a new pull request, #8748: URL: https://github.com/apache/iceberg/pull/8748 Add a missing semicolon to the "CREATE TABLE ..." statement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[PR] Update spark-getting-started.md [iceberg-docs]

2023-10-08 Thread via GitHub
Priyansh121096 opened a new pull request, #281: URL: https://github.com/apache/iceberg-docs/pull/281 Add a missing semicolon to the "CREATE TABLE ..." statement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Docs: Update spark-getting-started.md [iceberg]

2023-10-08 Thread via GitHub
Priyansh121096 commented on PR #8748: URL: https://github.com/apache/iceberg/pull/8748#issuecomment-1752168855 No worries @Fokko. Stumbled upon this while blindly copy pasting from the getting started page and thought I'd put in a fix. Btw, I've also raised https://github.com/apache/

[PR] Build: Bump pypa/cibuildwheel from 2.16.0 to 2.16.2 [iceberg-python]

2023-10-08 Thread via GitHub
dependabot[bot] opened a new pull request, #47: URL: https://github.com/apache/iceberg-python/pull/47 Bumps [pypa/cibuildwheel](https://github.com/pypa/cibuildwheel) from 2.16.0 to 2.16.2. Release notes Sourced from https://github.com/pypa/cibuildwheel/releases";>pypa/cibuildwheel'

[PR] Build: Bump cython from 3.0.2 to 3.0.3 [iceberg-python]

2023-10-08 Thread via GitHub
dependabot[bot] opened a new pull request, #48: URL: https://github.com/apache/iceberg-python/pull/48 Bumps [cython](https://github.com/cython/cython) from 3.0.2 to 3.0.3. Changelog Sourced from https://github.com/cython/cython/blob/master/CHANGES.rst";>cython's changelog. 3

[PR] Build: Bump psycopg2-binary from 2.9.8 to 2.9.9 [iceberg-python]

2023-10-08 Thread via GitHub
dependabot[bot] opened a new pull request, #49: URL: https://github.com/apache/iceberg-python/pull/49 Bumps [psycopg2-binary](https://github.com/psycopg/psycopg2) from 2.9.8 to 2.9.9. Changelog Sourced from https://github.com/psycopg/psycopg2/blob/master/NEWS";>psycopg2-binary's c

[PR] Build: Bump coverage from 7.3.1 to 7.3.2 [iceberg-python]

2023-10-08 Thread via GitHub
dependabot[bot] opened a new pull request, #50: URL: https://github.com/apache/iceberg-python/pull/50 Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.3.1 to 7.3.2. Changelog Sourced from https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst";>coverage's changel

[PR] Build: Bump fastavro from 1.8.3 to 1.8.4 [iceberg-python]

2023-10-08 Thread via GitHub
dependabot[bot] opened a new pull request, #51: URL: https://github.com/apache/iceberg-python/pull/51 Bumps [fastavro](https://github.com/fastavro/fastavro) from 1.8.3 to 1.8.4. Changelog Sourced from https://github.com/fastavro/fastavro/blob/master/ChangeLog";>fastavro's changelog

[PR] Build: Bump mkdocstrings-python from 1.7.1 to 1.7.2 [iceberg-python]

2023-10-08 Thread via GitHub
dependabot[bot] opened a new pull request, #52: URL: https://github.com/apache/iceberg-python/pull/52 Bumps [mkdocstrings-python](https://github.com/mkdocstrings/python) from 1.7.1 to 1.7.2. Release notes Sourced from https://github.com/mkdocstrings/python/releases";>mkdocstrings-p

Re: [I] We may not need to store partition columns in the datafiles [iceberg]

2023-10-08 Thread via GitHub
github-actions[bot] commented on issue #7225: URL: https://github.com/apache/iceberg/issues/7225#issuecomment-1752202170 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] We may not need to store partition columns in the datafiles [iceberg]

2023-10-08 Thread via GitHub
github-actions[bot] closed issue #7225: We may not need to store partition columns in the datafiles URL: https://github.com/apache/iceberg/issues/7225 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] pos delete files not removed immediately after rewrite [iceberg]

2023-10-08 Thread via GitHub
zinking closed issue #8743: pos delete files not removed immediately after rewrite URL: https://github.com/apache/iceberg/issues/8743 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

[PR] Flink:backport PR to 1.15 #7360: Implement data statistics coordinator to aggregate data statistics from operator subtasks [iceberg]

2023-10-08 Thread via GitHub
yegangy0718 opened a new pull request, #8749: URL: https://github.com/apache/iceberg/pull/8749 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[PR] Nessie: Remove dead code in NessieCatalog [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat opened a new pull request, #8750: URL: https://github.com/apache/iceberg/pull/8750 After https://github.com/apache/iceberg/pull/8607, `createNessieClientBuilder` is not used anymore in the code. So, it should be removed. -- This is an automated message from the Apache Gi

Re: [PR] Nessie: Remove dead code in NessieCatalog [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat commented on PR #8750: URL: https://github.com/apache/iceberg/pull/8750#issuecomment-1752372176 cc: @snazy, @dimas-b, @adutra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Docs: Update spark-getting-started.md [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat commented on code in PR #8748: URL: https://github.com/apache/iceberg/pull/8748#discussion_r1349878591 ## docs/spark-getting-started.md: ## @@ -69,7 +69,7 @@ To create your first Iceberg table in Spark, use the `spark-sql` shell or `spark ```sql -- local is the

Re: [PR] Docs: Update spark-getting-started.md [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat commented on PR #8748: URL: https://github.com/apache/iceberg/pull/8748#issuecomment-1752381404 > Btw, I've also raised https://github.com/apache/iceberg-docs/pull/281. I'm not sure which is the one that actually gets deployed to https://iceberg.apache.org/docs/latest/getting-s

Re: [PR] Spec: Clarify missing fields when writing [iceberg]

2023-10-08 Thread via GitHub
ajantha-bhat commented on code in PR #8672: URL: https://github.com/apache/iceberg/pull/8672#discussion_r1349879908 ## format/spec.md: ## @@ -128,13 +128,13 @@ Tables do not require rename, except for tables that use atomic rename to implem Writer requirements -Some t

Re: [PR] Dell : Migrate Files using TestRule to Junit5. [iceberg]

2023-10-08 Thread via GitHub
nastra commented on code in PR #8707: URL: https://github.com/apache/iceberg/pull/8707#discussion_r1349886171 ## build.gradle: ## @@ -912,7 +912,6 @@ project(':iceberg-dell') { implementation project(':iceberg-common') implementation project(path: ':iceberg-bundled-gua

Re: [PR] Dell : Migrate Files using TestRule to Junit5. [iceberg]

2023-10-08 Thread via GitHub
nastra commented on code in PR #8707: URL: https://github.com/apache/iceberg/pull/8707#discussion_r1349886330 ## dell/src/test/java/org/apache/iceberg/dell/mock/ecs/EcsS3MockRule.java: ## @@ -178,4 +163,16 @@ public String bucket() { public String randomObjectName() { re

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.18.0 to 26.24.0 [iceberg]

2023-10-08 Thread via GitHub
nastra merged PR #8735: URL: https://github.com/apache/iceberg/pull/8735 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Python: Remove python directory and references [iceberg]

2023-10-08 Thread via GitHub
nastra merged PR #8695: URL: https://github.com/apache/iceberg/pull/8695 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Build: Bump org.immutables:value from 2.9.2 to 2.10.0 [iceberg]

2023-10-08 Thread via GitHub
nastra commented on PR #8736: URL: https://github.com/apache/iceberg/pull/8736#issuecomment-1752404129 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To