Re: [PR] Spark 3.5: Add a procedure to remove corrupt snapshots [iceberg]

2024-02-04 Thread via GitHub
nastra commented on code in PR #9645: URL: https://github.com/apache/iceberg/pull/9645#discussion_r1477782650 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveSnapshotsProcedure.java: ## @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache So

Re: [PR] Spark 3.5: Add a procedure to remove corrupt snapshots [iceberg]

2024-02-04 Thread via GitHub
nastra commented on code in PR #9645: URL: https://github.com/apache/iceberg/pull/9645#discussion_r1477781944 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRemoveSnapshotsProcedure.java: ## @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache So

Re: [I] Use latest Parquet version for writing [iceberg-python]

2024-02-04 Thread via GitHub
jonashaag commented on issue #359: URL: https://github.com/apache/iceberg-python/issues/359#issuecomment-1926401364 @syun64 @Fokko @HonahX feedback? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Core: only trim slash when warehouse location is not root path [iceberg]

2024-02-04 Thread via GitHub
nastra commented on code in PR #9619: URL: https://github.com/apache/iceberg/pull/9619#discussion_r141865 ## core/src/test/java/org/apache/iceberg/util/TestLocationUtil.java: ## @@ -46,6 +46,21 @@ public void testStripTrailingSlash() { assertThat(LocationUtil.stripTrail

Re: [PR] Core: only trim slash when warehouse location is not root path [iceberg]

2024-02-04 Thread via GitHub
nastra commented on code in PR #9619: URL: https://github.com/apache/iceberg/pull/9619#discussion_r141865 ## core/src/test/java/org/apache/iceberg/util/TestLocationUtil.java: ## @@ -46,6 +46,21 @@ public void testStripTrailingSlash() { assertThat(LocationUtil.stripTrail

Re: [I] DOCS: Report CSS and styling issues on the new site. [iceberg]

2024-02-04 Thread via GitHub
manuzhang commented on issue #9643: URL: https://github.com/apache/iceberg/issues/9643#issuecomment-1926369554 https://github.com/apache/iceberg-docs/tree/main/iceberg-theme/static/lottie is missing. For example, ![CleanShot 2024-02-05 at 15 24 31](https://github.com/apache/iceberg/a

Re: [PR] Bug Fix: Allow GlueCatalog to create table with TimestampzType [iceberg-python]

2024-02-04 Thread via GitHub
HonahX commented on code in PR #366: URL: https://github.com/apache/iceberg-python/pull/366#discussion_r1477752559 ## pyiceberg/catalog/glue.py: ## @@ -150,7 +152,7 @@ def primitive(self, primitive: PrimitiveType) -> str: if isinstance(primitive, DecimalType):

Re: [PR] Bug Fix: Allow GlueCatalog to create table with TimestampzType [iceberg-python]

2024-02-04 Thread via GitHub
HonahX commented on code in PR #366: URL: https://github.com/apache/iceberg-python/pull/366#discussion_r1477752559 ## pyiceberg/catalog/glue.py: ## @@ -150,7 +152,7 @@ def primitive(self, primitive: PrimitiveType) -> str: if isinstance(primitive, DecimalType):

Re: [PR] Bug Fix: Allow GlueCatalog to create table with TimestampzType [iceberg-python]

2024-02-04 Thread via GitHub
HonahX commented on code in PR #366: URL: https://github.com/apache/iceberg-python/pull/366#discussion_r1477752559 ## pyiceberg/catalog/glue.py: ## @@ -150,7 +152,7 @@ def primitive(self, primitive: PrimitiveType) -> str: if isinstance(primitive, DecimalType):

[PR] Bug Fix: Allow GlueCatalog to create table with TimestampzType [iceberg-python]

2024-02-04 Thread via GitHub
HonahX opened a new pull request, #366: URL: https://github.com/apache/iceberg-python/pull/366 Pyiceberg uses `TimestampType` for timestamp without time zone and `TimestampzType` for timestamp with time zone. This PR adds the missing conversion from `TimestampzType` to glue type str

Re: [PR] Spark 3.5: Add an option not to delete files in ExpireSnapshots [iceberg]

2024-02-04 Thread via GitHub
manuzhang commented on PR #9584: URL: https://github.com/apache/iceberg/pull/9584#issuecomment-1926335699 I've created https://github.com/apache/iceberg/pull/9645, adding `RemoveSnapshotsProcedure` to remove corrupt snapshots. -- This is an automated message from the Apache Git Service. T

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-04 Thread via GitHub
jbonofre commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1477725114 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -80,19 +83,37 @@ public class JdbcCatalog extends BaseMetastoreCatalog private final Function,

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-02-04 Thread via GitHub
liurenjie1024 commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1477719264 ## crates/iceberg/src/writer/file_writer/location_generator.rs: ## @@ -0,0 +1,108 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] Partition Evolution [iceberg-python]

2024-02-04 Thread via GitHub
amogh-jahagirdar commented on code in PR #245: URL: https://github.com/apache/iceberg-python/pull/245#discussion_r1477705066 ## pyiceberg/table/__init__.py: ## @@ -868,6 +919,12 @@ def sort_orders(self) -> Dict[int, SortOrder]: """Return a dict of the sort orders of thi

Re: [PR] Partition Evolution [iceberg-python]

2024-02-04 Thread via GitHub
amogh-jahagirdar commented on code in PR #245: URL: https://github.com/apache/iceberg-python/pull/245#discussion_r1477706261 ## pyiceberg/table/__init__.py: ## @@ -533,6 +551,39 @@ def _(update: SetCurrentSchemaUpdate, base_metadata: TableMetadata, context: _Ta return base

Re: [PR] Partition Evolution [iceberg-python]

2024-02-04 Thread via GitHub
amogh-jahagirdar commented on code in PR #245: URL: https://github.com/apache/iceberg-python/pull/245#discussion_r1477705066 ## pyiceberg/table/__init__.py: ## @@ -868,6 +919,12 @@ def sort_orders(self) -> Dict[int, SortOrder]: """Return a dict of the sort orders of thi

Re: [PR] Partition Evolution [iceberg-python]

2024-02-04 Thread via GitHub
HonahX commented on code in PR #245: URL: https://github.com/apache/iceberg-python/pull/245#discussion_r1477693599 ## tests/test_integration_partition_evolution.py: ## @@ -0,0 +1,423 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Partition Evolution [iceberg-python]

2024-02-04 Thread via GitHub
HonahX commented on code in PR #245: URL: https://github.com/apache/iceberg-python/pull/245#discussion_r1477690350 ## pyiceberg/table/__init__.py: ## @@ -533,6 +551,39 @@ def _(update: SetCurrentSchemaUpdate, base_metadata: TableMetadata, context: _Ta return base_metadata.

Re: [PR] Partition Evolution [iceberg-python]

2024-02-04 Thread via GitHub
HonahX commented on code in PR #245: URL: https://github.com/apache/iceberg-python/pull/245#discussion_r1477690350 ## pyiceberg/table/__init__.py: ## @@ -533,6 +551,39 @@ def _(update: SetCurrentSchemaUpdate, base_metadata: TableMetadata, context: _Ta return base_metadata.

Re: [PR] Partition Evolution [iceberg-python]

2024-02-04 Thread via GitHub
HonahX commented on code in PR #245: URL: https://github.com/apache/iceberg-python/pull/245#discussion_r1477690350 ## pyiceberg/table/__init__.py: ## @@ -533,6 +551,39 @@ def _(update: SetCurrentSchemaUpdate, base_metadata: TableMetadata, context: _Ta return base_metadata.

Re: [I] DOCS: Report CSS and styling issues on the new site. [iceberg]

2024-02-04 Thread via GitHub
bitsondatadev commented on issue #9643: URL: https://github.com/apache/iceberg/issues/9643#issuecomment-1926233652 The nav font seems really small ![image (3)](https://github.com/apache/iceberg/assets/8547669/c85a056a-a1cd-4146-9ef4-69f7f257c760) url - releases browser - Chr

Re: [I] DOCS: Report CSS and styling issues on the new site. [iceberg]

2024-02-04 Thread via GitHub
bitsondatadev commented on issue #9643: URL: https://github.com/apache/iceberg/issues/9643#issuecomment-1926230554 Seems like bulleted lists have a lot of extra space. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [I] DOCS: Report CSS and styling issues on the new site. [iceberg]

2024-02-04 Thread via GitHub
bitsondatadev commented on issue #9643: URL: https://github.com/apache/iceberg/issues/9643#issuecomment-1926224706 On the mobile site, the buttons and heders aren't displaying well. url - homepage browser - Firefox browser version - 122.0 computer platform -iPhone --

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-02-04 Thread via GitHub
ZENOTME commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1477655462 ## crates/iceberg/src/writer/file_writer/location_generator.rs: ## @@ -0,0 +1,108 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contr

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-02-04 Thread via GitHub
ZENOTME commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1477651653 ## crates/iceberg/src/writer/file_writer/location_generator.rs: ## @@ -0,0 +1,108 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contr

Re: [I] Hive: partitioning is not working [iceberg]

2024-02-04 Thread via GitHub
bluzy commented on issue #9329: URL: https://github.com/apache/iceberg/issues/9329#issuecomment-1926160438 I guess nested column seems to be cause. When I tested with 1-depth partition column, then the problem is not occured. -- This is an automated message from the Apache Git Service.

Re: [PR] Add UnboundSortOrder [iceberg-rust]

2024-02-04 Thread via GitHub
liurenjie1024 commented on PR #115: URL: https://github.com/apache/iceberg-rust/pull/115#issuecomment-1926133182 cc @Fokko Could we merge this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] DOCS: Report CSS and styling issues on the new site. [iceberg]

2024-02-04 Thread via GitHub
bitsondatadev commented on issue #9643: URL: https://github.com/apache/iceberg/issues/9643#issuecomment-1926123621 @manuzhang, this looks like a caching issue. I recommend you clear all cache for the Iceberg site, especially if you frequent it. -- This is an automated message from the Apa

Re: [PR] Iceberg site fixes [iceberg]

2024-02-04 Thread via GitHub
bitsondatadev commented on code in PR #9642: URL: https://github.com/apache/iceberg/pull/9642#discussion_r1477580106 ## site/nav.yml: ## @@ -32,18 +32,18 @@ nav: - Talks: talks.md - Vendors: vendors.md - Project: -- Join: community.md Review Comment: 😎 I've wan

Re: [PR] Core: HadoopTable needs to skip file cleanup after task failure under some boundary conditions. [iceberg]

2024-02-04 Thread via GitHub
BsoBird commented on PR #9546: URL: https://github.com/apache/iceberg/pull/9546#issuecomment-1926106692 If the user's fs.rename call is taking too long (slow file system response) and the user has set the metadata-ttl to be too short, then it is possible that the user will commit an older v

Re: [PR] Iceberg site fixes [iceberg]

2024-02-04 Thread via GitHub
bitsondatadev commented on code in PR #9642: URL: https://github.com/apache/iceberg/pull/9642#discussion_r1477580106 ## site/nav.yml: ## @@ -32,18 +32,18 @@ nav: - Talks: talks.md - Vendors: vendors.md - Project: -- Join: community.md Review Comment: 😎 --

Re: [PR] Core: Add view support for JDBC catalog [iceberg]

2024-02-04 Thread via GitHub
ajantha-bhat commented on code in PR #9487: URL: https://github.com/apache/iceberg/pull/9487#discussion_r1477365303 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java: ## @@ -80,19 +83,37 @@ public class JdbcCatalog extends BaseMetastoreCatalog private final Funct

Re: [PR] Update blogs.md [iceberg]

2024-02-04 Thread via GitHub
Fokko merged PR #9552: URL: https://github.com/apache/iceberg/pull/9552 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

Re: [PR] Update blogs.md [iceberg]

2024-02-04 Thread via GitHub
ajantha-bhat commented on PR #9552: URL: https://github.com/apache/iceberg/pull/9552#issuecomment-1926094048 I think this can be merged now. cc: @nastra, @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] feat: Bump version 0.2.0 to prepare for release. [iceberg-rust]

2024-02-04 Thread via GitHub
liurenjie1024 commented on PR #181: URL: https://github.com/apache/iceberg-rust/pull/181#issuecomment-1926090864 > @liurenjie1024 Now #174 has been merged, can you regenerate the dependencies tsv's? Yeah, let's move! -- This is an automated message from the Apache Git Service. To r

Re: [PR] refactor: rm async_trait and add trait_variant [iceberg-rust]

2024-02-04 Thread via GitHub
liurenjie1024 commented on code in PR #186: URL: https://github.com/apache/iceberg-rust/pull/186#discussion_r1477570301 ## crates/iceberg/src/catalog/mod.rs: ## @@ -25,16 +25,16 @@ use crate::spec::{ }; use crate::table::Table; use crate::{Error, ErrorKind, Result}; -use asyn

Re: [PR] feat: add parquet writer [iceberg-rust]

2024-02-04 Thread via GitHub
liurenjie1024 commented on code in PR #176: URL: https://github.com/apache/iceberg-rust/pull/176#discussion_r1477561649 ## crates/iceberg/src/writer/file_writer/location_generator.rs: ## @@ -0,0 +1,108 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [I] DOCS: Report CSS and styling issues on the new site. [iceberg]

2024-02-04 Thread via GitHub
manuzhang commented on issue #9643: URL: https://github.com/apache/iceberg/issues/9643#issuecomment-1926084089 https://iceberg.apache.org/docs/latest/configuration/ on Firefox 122.0, M1 Mac 12.7.2 ![CleanShot 2024-02-05 at 09 49 49](https://github.com/apache/iceberg/assets/1191767/51eecd

Re: [PR] refactor: rm async_trait and add trait_variant [iceberg-rust]

2024-02-04 Thread via GitHub
odysa commented on code in PR #186: URL: https://github.com/apache/iceberg-rust/pull/186#discussion_r1477561928 ## crates/iceberg/src/catalog/mod.rs: ## @@ -25,16 +25,16 @@ use crate::spec::{ }; use crate::table::Table; use crate::{Error, ErrorKind, Result}; -use async_trait:

Re: [PR] feat: Bump version 0.2.0 to prepare for release. [iceberg-rust]

2024-02-04 Thread via GitHub
Fokko commented on PR #181: URL: https://github.com/apache/iceberg-rust/pull/181#issuecomment-1926075085 @liurenjie1024 Now #174 has been merged, can you regenerate the dependencies tsv's? -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] feat: Bump hive_metastore to use pure rust thrift impl `volo` [iceberg-rust]

2024-02-04 Thread via GitHub
Fokko merged PR #174: URL: https://github.com/apache/iceberg-rust/pull/174 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] Fix: avro schema names for manifest and manifest_list [iceberg-rust]

2024-02-04 Thread via GitHub
Fokko merged PR #182: URL: https://github.com/apache/iceberg-rust/pull/182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [I] Change avro schema names for `ManifestList` and `Manifest` to the official ones [iceberg-rust]

2024-02-04 Thread via GitHub
Fokko closed issue #179: Change avro schema names for `ManifestList` and `Manifest` to the official ones URL: https://github.com/apache/iceberg-rust/issues/179 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] feat: add handwritten serialize [iceberg-rust]

2024-02-04 Thread via GitHub
Fokko merged PR #185: URL: https://github.com/apache/iceberg-rust/pull/185 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.ap

Re: [PR] refactor: rm async_trait and add trait_variant [iceberg-rust]

2024-02-04 Thread via GitHub
liurenjie1024 commented on code in PR #186: URL: https://github.com/apache/iceberg-rust/pull/186#discussion_r1477552350 ## crates/iceberg/src/catalog/mod.rs: ## @@ -25,16 +25,16 @@ use crate::spec::{ }; use crate::table::Table; use crate::{Error, ErrorKind, Result}; -use asyn

Re: [I] Pig should not rely on it own implementation of InputFormat [iceberg]

2024-02-04 Thread via GitHub
github-actions[bot] commented on issue #847: URL: https://github.com/apache/iceberg/issues/847#issuecomment-1926004065 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Add support for reading Avro and ORC data files in Iceberg Pig [iceberg]

2024-02-04 Thread via GitHub
github-actions[bot] commented on issue #849: URL: https://github.com/apache/iceberg/issues/849#issuecomment-1926004106 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Vectorized reads - enable vectorized reads when projecting identity partition columns [iceberg]

2024-02-04 Thread via GitHub
github-actions[bot] commented on issue #838: URL: https://github.com/apache/iceberg/issues/838#issuecomment-1926004029 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] AvroFileAppender doesn't report column-level stats [iceberg]

2024-02-04 Thread via GitHub
github-actions[bot] commented on issue #837: URL: https://github.com/apache/iceberg/issues/837#issuecomment-1926004010 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. T

Re: [I] Support case-sensitivity in all operations with metrics evaluators [iceberg]

2024-02-04 Thread via GitHub
github-actions[bot] closed issue #413: Support case-sensitivity in all operations with metrics evaluators URL: https://github.com/apache/iceberg/issues/413 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Support case-sensitivity in all operations with metrics evaluators [iceberg]

2024-02-04 Thread via GitHub
github-actions[bot] commented on issue #413: URL: https://github.com/apache/iceberg/issues/413#issuecomment-1926003865 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git

[I] Implement Centralized Management of Table Properties [iceberg-python]

2024-02-04 Thread via GitHub
HonahX opened a new issue, #365: URL: https://github.com/apache/iceberg-python/issues/365 ### Feature Request / Improvement With the release of version 0.6.0, pyiceberg is set to expand its capabilities in handling a greater variety of table properties, encompassing both read and wri

Re: [PR] docs: Document Parquet write options [iceberg-python]

2024-02-04 Thread via GitHub
HonahX commented on code in PR #364: URL: https://github.com/apache/iceberg-python/pull/364#discussion_r1477469426 ## mkdocs/docs/configuration.md: ## @@ -62,7 +62,16 @@ You can also set the FileIO explicitly: | -- | | -

Re: [PR] docs: Document Parquet write options [iceberg-python]

2024-02-04 Thread via GitHub
HonahX commented on code in PR #364: URL: https://github.com/apache/iceberg-python/pull/364#discussion_r1477469426 ## mkdocs/docs/configuration.md: ## @@ -62,7 +62,16 @@ You can also set the FileIO explicitly: | -- | | -

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
HonahX commented on code in PR #358: URL: https://github.com/apache/iceberg-python/pull/358#discussion_r1477467681 ## pyiceberg/io/pyarrow.py: ## @@ -1745,14 +1747,42 @@ def write_file(table: Table, tasks: Iterator[WriteTask]) -> Iterator[DataFile]: key_metadata=None,

[PR] docs: Document Parquet write options [iceberg-python]

2024-02-04 Thread via GitHub
Fokko opened a new pull request, #364: URL: https://github.com/apache/iceberg-python/pull/364 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] fix postgres catalog initialization when tables do not exist [iceberg-python]

2024-02-04 Thread via GitHub
Fokko merged PR #356: URL: https://github.com/apache/iceberg-python/pull/356 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] SqlCatalog, default create_engine echo to False [iceberg-python]

2024-02-04 Thread via GitHub
Fokko merged PR #360: URL: https://github.com/apache/iceberg-python/pull/360 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] fix postgres catalog initialization when tables do not exist [iceberg-python]

2024-02-04 Thread via GitHub
Fokko commented on code in PR #356: URL: https://github.com/apache/iceberg-python/pull/356#discussion_r1477447763 ## pyiceberg/catalog/sql.py: ## @@ -111,7 +111,7 @@ def _ensure_tables_exist(self) -> None: stmt = select(1).select_from(table) try

Re: [PR] Get Started: Add sqlcatalog and local fs warehouse [iceberg-python]

2024-02-04 Thread via GitHub
Fokko merged PR #361: URL: https://github.com/apache/iceberg-python/pull/361 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
jonashaag commented on code in PR #358: URL: https://github.com/apache/iceberg-python/pull/358#discussion_r1477447145 ## pyiceberg/io/pyarrow.py: ## @@ -1745,14 +1747,42 @@ def write_file(table: Table, tasks: Iterator[WriteTask]) -> Iterator[DataFile]: key_metadata=Non

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
Fokko commented on code in PR #358: URL: https://github.com/apache/iceberg-python/pull/358#discussion_r1477443042 ## pyiceberg/io/pyarrow.py: ## @@ -1745,14 +1747,42 @@ def write_file(table: Table, tasks: Iterator[WriteTask]) -> Iterator[DataFile]: key_metadata=None,

Re: [PR] Add Daft examples and code into PyIceberg docs and Table [iceberg-python]

2024-02-04 Thread via GitHub
Fokko commented on PR #355: URL: https://github.com/apache/iceberg-python/pull/355#issuecomment-1925918588 Should we also have some sanity checks, for example: https://github.com/apache/iceberg-python/blob/a4856bc2eadf90ac85dec96d4502ca3517bb1bb5/tests/integration/test_reads.py#L184 -- Th

Re: [PR] Get Started: Add sqlcatalog and local fs warehouse [iceberg-python]

2024-02-04 Thread via GitHub
kevinjqliu commented on code in PR #361: URL: https://github.com/apache/iceberg-python/pull/361#discussion_r1477439057 ## mkdocs/docs/index.md: ## @@ -62,6 +62,27 @@ You either need to install `s3fs`, `adlfs`, `gcs`, or `pyarrow` to be able to fe Iceberg leverages the [catal

Re: [PR] Get Started: Add sqlcatalog and local fs warehouse [iceberg-python]

2024-02-04 Thread via GitHub
kevinjqliu commented on PR #361: URL: https://github.com/apache/iceberg-python/pull/361#issuecomment-1925915940 thanks for the review @Fokko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Improve error message in case of a mismatch [iceberg-python]

2024-02-04 Thread via GitHub
HonahX commented on code in PR #352: URL: https://github.com/apache/iceberg-python/pull/352#discussion_r1477438386 ## pyiceberg/table/__init__.py: ## @@ -133,6 +132,41 @@ _JAVA_LONG_MAX = 9223372036854775807 +def _check_schema(table_schema: Schema, other_schema: "pa.Schema"

Re: [PR] feat: add handwritten serialize [iceberg-rust]

2024-02-04 Thread via GitHub
odysa commented on PR #185: URL: https://github.com/apache/iceberg-rust/pull/185#issuecomment-1925914243 Replaced `expect` with `ok_or` in this PR as suggested. cc @liurenjie1024 @ZENOTME -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Get Started: Add sqlcatalog and local fs warehouse [iceberg-python]

2024-02-04 Thread via GitHub
Fokko commented on code in PR #361: URL: https://github.com/apache/iceberg-python/pull/361#discussion_r1477436914 ## mkdocs/docs/index.md: ## @@ -158,6 +177,14 @@ df = table.scan(row_filter="tip_per_mile > 0").to_arrow() len(df) ``` +### Explore Iceberg data and metadata fil

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
jonashaag commented on PR #358: URL: https://github.com/apache/iceberg-python/pull/358#issuecomment-1925911513 Sweet, ready to merge from my POV -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Add Daft examples and code into PyIceberg docs and Table [iceberg-python]

2024-02-04 Thread via GitHub
Fokko commented on code in PR #355: URL: https://github.com/apache/iceberg-python/pull/355#discussion_r1477436119 ## pyproject.toml: ## @@ -105,6 +105,7 @@ pyarrow = ["pyarrow"] pandas = ["pandas", "pyarrow"] duckdb = ["duckdb", "pyarrow"] ray = ["ray", "pyarrow", "pandas"] +

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
jonashaag commented on code in PR #358: URL: https://github.com/apache/iceberg-python/pull/358#discussion_r1477435123 ## pyiceberg/io/pyarrow.py: ## @@ -1745,14 +1747,41 @@ def write_file(table: Table, tasks: Iterator[WriteTask]) -> Iterator[DataFile]: key_metadata=Non

Re: [PR] SqlCatalog, default create_engine echo to False [iceberg-python]

2024-02-04 Thread via GitHub
Fokko commented on code in PR #360: URL: https://github.com/apache/iceberg-python/pull/360#discussion_r1477434800 ## pyiceberg/catalog/sql.py: ## @@ -101,7 +101,8 @@ def __init__(self, name: str, **properties: str): if not (uri_prop := self.properties.get("uri")):

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
Fokko commented on code in PR #358: URL: https://github.com/apache/iceberg-python/pull/358#discussion_r1477434938 ## tests/integration/test_writes.py: ## @@ -489,6 +492,50 @@ def test_data_files(spark: SparkSession, session_catalog: Catalog, arrow_table_w assert [row.delet

Re: [PR] fix postgres catalog initialization when tables do not exist [iceberg-python]

2024-02-04 Thread via GitHub
Fokko commented on PR #356: URL: https://github.com/apache/iceberg-python/pull/356#issuecomment-1925907114 @rahij Sure thing! Can you run `make lint` to fix the style issues? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
Fokko commented on PR #358: URL: https://github.com/apache/iceberg-python/pull/358#issuecomment-1925904312 > Sorry I don't feel comfortable writing documentation because I still lack a lot of Iceberg understanding and terminology. Could you do that part please? Sure thing, no problem

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
jonashaag commented on PR #358: URL: https://github.com/apache/iceberg-python/pull/358#issuecomment-1925898463 Sorry I don't feel comfortable writing documentation because I still lack a lot of Iceberg understanding and terminology. Could you do that part please? -- This is an automated m

Re: [PR] Iceberg site fixes [iceberg]

2024-02-04 Thread via GitHub
Fokko commented on code in PR #9642: URL: https://github.com/apache/iceberg/pull/9642#discussion_r1477429039 ## site/nav.yml: ## @@ -32,18 +32,18 @@ nav: - Talks: talks.md - Vendors: vendors.md - Project: -- Join: community.md Review Comment: I like this 👍

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
jonashaag commented on code in PR #358: URL: https://github.com/apache/iceberg-python/pull/358#discussion_r1477429231 ## pyiceberg/io/pyarrow.py: ## @@ -1745,14 +1747,41 @@ def write_file(table: Table, tasks: Iterator[WriteTask]) -> Iterator[DataFile]: key_metadata=Non

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
jonashaag commented on code in PR #358: URL: https://github.com/apache/iceberg-python/pull/358#discussion_r1477429043 ## pyiceberg/io/pyarrow.py: ## @@ -1745,14 +1747,41 @@ def write_file(table: Table, tasks: Iterator[WriteTask]) -> Iterator[DataFile]: key_metadata=Non

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
Fokko commented on code in PR #358: URL: https://github.com/apache/iceberg-python/pull/358#discussion_r1477428773 ## pyiceberg/io/pyarrow.py: ## @@ -1745,14 +1747,41 @@ def write_file(table: Table, tasks: Iterator[WriteTask]) -> Iterator[DataFile]: key_metadata=None,

[PR] Flink: Overwrite All Partitions [iceberg]

2024-02-04 Thread via GitHub
aschreiber1 opened a new pull request, #9644: URL: https://github.com/apache/iceberg/pull/9644 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[I] DOCS: Report CSS and styling issues on the new site. [iceberg]

2024-02-04 Thread via GitHub
bitsondatadev opened a new issue, #9643: URL: https://github.com/apache/iceberg/issues/9643 Please use this issue to report various CSS and styling issues that we need to address over time. You can add to this by adding some of the following information and any other relevant information:

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
jonashaag commented on PR #358: URL: https://github.com/apache/iceberg-python/pull/358#issuecomment-1925891201 I've changed the properties to be table properties and added handling for some other Parquet properites -- This is an automated message from the Apache Git Service. To respond to

[PR] Iceberg site fixes [iceberg]

2024-02-04 Thread via GitHub
bitsondatadev opened a new pull request, #9642: URL: https://github.com/apache/iceberg/pull/9642 # Move catalog page under concepts header https://github.com/apache/iceberg/assets/8547669/4fc6e502-ce39-4916-af1e-0c985add100d";> Getting a 401 "Not Found" when I go to the link cac

Re: [PR] Docs: Enhance Java quickstart example [iceberg]

2024-02-04 Thread via GitHub
rdblue commented on PR #9585: URL: https://github.com/apache/iceberg/pull/9585#issuecomment-1925849381 Thanks, @manuzhang! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Docs: Enhance Java quickstart example [iceberg]

2024-02-04 Thread via GitHub
rdblue merged PR #9585: URL: https://github.com/apache/iceberg/pull/9585 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] refactor: rm async_trait and add trait_variant [iceberg-rust]

2024-02-04 Thread via GitHub
odysa commented on code in PR #186: URL: https://github.com/apache/iceberg-rust/pull/186#discussion_r1477382004 ## crates/iceberg/src/catalog/mod.rs: ## @@ -25,16 +25,16 @@ use crate::spec::{ }; use crate::table::Table; use crate::{Error, ErrorKind, Result}; -use async_trait:

[PR] Kafka Connect: Record converters and delta writers [iceberg]

2024-02-04 Thread via GitHub
bryanck opened a new pull request, #9641: URL: https://github.com/apache/iceberg/pull/9641 This PR is the next stage in submitting the Iceberg Kafka Connect sink connector, and is a follow up to https://github.com/apache/iceberg/pull/8701 and https://github.com/apache/iceberg/pull/9466. It

Re: [PR] Support usage of Separate OIDC Authorization Server URI [iceberg]

2024-02-04 Thread via GitHub
syun64 commented on code in PR #8976: URL: https://github.com/apache/iceberg/pull/8976#discussion_r1477366640 ## core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java: ## @@ -361,6 +392,23 @@ public static class AuthSession { private final String credential;

Re: [I] iceberg-mr: Switch class level parameterized tests to Junit5 [iceberg]

2024-02-04 Thread via GitHub
lisirrx commented on issue #9269: URL: https://github.com/apache/iceberg/issues/9269#issuecomment-1925821859 hi @nastra , could you please assign this task to me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] feat: add handwritten serialize [iceberg-rust]

2024-02-04 Thread via GitHub
odysa commented on code in PR #185: URL: https://github.com/apache/iceberg-rust/pull/185#discussion_r1477362498 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -676,9 +690,10 @@ pub(super) mod _serde { } } -impl From for TableMetadataV1 { -fn from

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
syun64 commented on code in PR #358: URL: https://github.com/apache/iceberg-python/pull/358#discussion_r1477330747 ## tests/integration/test_writes.py: ## @@ -489,6 +492,58 @@ def test_data_files(spark: SparkSession, session_catalog: Catalog, arrow_table_w assert [row.dele

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
syun64 commented on code in PR #358: URL: https://github.com/apache/iceberg-python/pull/358#discussion_r1477316186 ## pyiceberg/io/pyarrow.py: ## @@ -1720,13 +1720,22 @@ def write_file(table: Table, tasks: Iterator[WriteTask]) -> Iterator[DataFile]: except StopIteration:

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
syun64 commented on code in PR #358: URL: https://github.com/apache/iceberg-python/pull/358#discussion_r1477316186 ## pyiceberg/io/pyarrow.py: ## @@ -1720,13 +1720,22 @@ def write_file(table: Table, tasks: Iterator[WriteTask]) -> Iterator[DataFile]: except StopIteration:

Re: [PR] feat: add handwritten serialize [iceberg-rust]

2024-02-04 Thread via GitHub
ZENOTME commented on code in PR #185: URL: https://github.com/apache/iceberg-rust/pull/185#discussion_r1477320178 ## crates/iceberg/src/spec/table_metadata.rs: ## @@ -676,9 +690,10 @@ pub(super) mod _serde { } } -impl From for TableMetadataV1 { -fn fr

Re: [PR] Kafka Connect: Sink connector with data writers and converters [iceberg]

2024-02-04 Thread via GitHub
ajantha-bhat commented on code in PR #9466: URL: https://github.com/apache/iceberg/pull/9466#discussion_r1477293784 ## kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/IcebergSinkConfig.java: ## @@ -0,0 +1,468 @@ +/* + * Licensed to the Apache Software Founda

Re: [PR] Use `write.parquet.compression-{codec,level}` [iceberg-python]

2024-02-04 Thread via GitHub
syun64 commented on code in PR #358: URL: https://github.com/apache/iceberg-python/pull/358#discussion_r1477316186 ## pyiceberg/io/pyarrow.py: ## @@ -1720,13 +1720,22 @@ def write_file(table: Table, tasks: Iterator[WriteTask]) -> Iterator[DataFile]: except StopIteration:

Re: [PR] feat: Bump hive_metastore to use pure rust thrift impl `volo` [iceberg-rust]

2024-02-04 Thread via GitHub
Xuanwo commented on PR #174: URL: https://github.com/apache/iceberg-rust/pull/174#issuecomment-1925703005 cc @liurenjie1024, please review again. Sorry for the late. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] feat: Bump hive_metastore to use pure rust thrift impl `volo` [iceberg-rust]

2024-02-04 Thread via GitHub
Xuanwo commented on code in PR #174: URL: https://github.com/apache/iceberg-rust/pull/174#discussion_r1477279307 ## crates/catalog/hms/src/catalog.rs: ## @@ -71,19 +54,29 @@ impl Debug for HmsCatalog { impl HmsCatalog { /// Create a new hms catalog. pub fn new(config:

Re: [PR] feat: Bump hive_metastore to use pure rust thrift impl `volo` [iceberg-rust]

2024-02-04 Thread via GitHub
Xuanwo commented on code in PR #174: URL: https://github.com/apache/iceberg-rust/pull/174#discussion_r1477279307 ## crates/catalog/hms/src/catalog.rs: ## @@ -71,19 +54,29 @@ impl Debug for HmsCatalog { impl HmsCatalog { /// Create a new hms catalog. pub fn new(config:

  1   2   >