[GitHub] [iceberg] nastra commented on a diff in pull request #6353: Make sure S3 stream opened by ReadConf ctor is closed

2022-12-09 Thread GitBox
nastra commented on code in PR #6353: URL: https://github.com/apache/iceberg/pull/6353#discussion_r1042500106 ## parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java: ## @@ -68,7 +72,7 @@ public ParquetReader( private ReadConf init() { if (conf == null)

[GitHub] [iceberg] Fokko merged pull request #6390: Build: Fix ./gradlew refreshJavadoc

2022-12-09 Thread GitBox
Fokko merged PR #6390: URL: https://github.com/apache/iceberg/pull/6390 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] Fokko commented on pull request #6390: Build: Fix ./gradlew refreshJavadoc

2022-12-09 Thread GitBox
Fokko commented on PR #6390: URL: https://github.com/apache/iceberg/pull/6390#issuecomment-1343999178 Thanks for fixing this @gaborkaszab and thanks @nastra and @singhpk234 for the review 👏🏻 -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [iceberg] dependabot[bot] opened a new pull request, #6396: Build: Bump certifi from 2022.9.24 to 2022.12.7 in /python

2022-12-09 Thread GitBox
dependabot[bot] opened a new pull request, #6396: URL: https://github.com/apache/iceberg/pull/6396 Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.9.24 to 2022.12.7. Commits https://github.com/certifi/python-certifi/commit/9e9e840925d7b8e76c76fdac1fab7e6e88

[GitHub] [iceberg] rajarshisarkar commented on a diff in pull request #6376: Docs: Add register table Spark procedure documentation

2022-12-09 Thread GitBox
rajarshisarkar commented on code in PR #6376: URL: https://github.com/apache/iceberg/pull/6376#discussion_r1044234964 ## docs/spark-procedures.md: ## @@ -493,6 +493,38 @@ CALL spark_catalog.system.add_files( ) ``` +### `register_table` + +Creates a catalog entry for a metada

[GitHub] [iceberg] rajarshisarkar commented on a diff in pull request #6376: Docs: Add register table Spark procedure documentation

2022-12-09 Thread GitBox
rajarshisarkar commented on code in PR #6376: URL: https://github.com/apache/iceberg/pull/6376#discussion_r1044235368 ## docs/spark-procedures.md: ## @@ -493,6 +493,38 @@ CALL spark_catalog.system.add_files( ) ``` +### `register_table` + +Creates a catalog entry for a metada

[GitHub] [iceberg] rajarshisarkar commented on a diff in pull request #6376: Docs: Add register table Spark procedure documentation

2022-12-09 Thread GitBox
rajarshisarkar commented on code in PR #6376: URL: https://github.com/apache/iceberg/pull/6376#discussion_r1044236371 ## docs/spark-procedures.md: ## @@ -493,6 +493,38 @@ CALL spark_catalog.system.add_files( ) ``` +### `register_table` + +Creates a catalog entry for a metada

[GitHub] [iceberg] rajarshisarkar commented on a diff in pull request #6376: Docs: Add register table Spark procedure documentation

2022-12-09 Thread GitBox
rajarshisarkar commented on code in PR #6376: URL: https://github.com/apache/iceberg/pull/6376#discussion_r1044243384 ## docs/spark-procedures.md: ## @@ -493,6 +493,38 @@ CALL spark_catalog.system.add_files( ) ``` +### `register_table` + +Creates a catalog entry for a metada

[GitHub] [iceberg] rajarshisarkar commented on pull request #6376: Docs: Add register table Spark procedure documentation

2022-12-09 Thread GitBox
rajarshisarkar commented on PR #6376: URL: https://github.com/apache/iceberg/pull/6376#issuecomment-1344055332 @RussellSpitzer I was able to generate the warning block with `{{< hint warning >}} ... {{< /hint >}}` and not with `!!! Warning`. I have attached the screenshot in the PR. Please

[GitHub] [iceberg-docs] gaborkaszab opened a new pull request, #187: Update the how-to-release page with findings after being a release manager

2022-12-09 Thread GitBox
gaborkaszab opened a new pull request, #187: URL: https://github.com/apache/iceberg-docs/pull/187 There are some obsolete steps, plus some that weren't on the list like revapi to run on the new version, etc. Also I found some useful information on different Apache projects how-to-release

[GitHub] [iceberg-docs] gaborkaszab commented on pull request #187: Update the how-to-release page with findings after being a release manager

2022-12-09 Thread GitBox
gaborkaszab commented on PR #187: URL: https://github.com/apache/iceberg-docs/pull/187#issuecomment-1344127711 cc @Fokko @ajantha-bhat -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [iceberg-docs] ajantha-bhat commented on a diff in pull request #187: Update the how-to-release page with findings after being a release manager

2022-12-09 Thread GitBox
ajantha-bhat commented on code in PR #187: URL: https://github.com/apache/iceberg-docs/pull/187#discussion_r1044318909 ## landing-page/content/common/how-to-release.md: ## @@ -222,6 +246,12 @@ Therefore, the release candidate is passed/rejected. After the release vote has pas

[GitHub] [iceberg] hililiwei commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
hililiwei commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044323786 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -308,6 +339,17 @@ public Scan buildChangelogScan() { return new Sp

[GitHub] [iceberg] Fokko commented on issue #6347: [Docs]: improve ChangeLog

2022-12-09 Thread GitBox
Fokko commented on issue #6347: URL: https://github.com/apache/iceberg/issues/6347#issuecomment-1344171422 Hey @InvisibleProgrammer I like the idea. The changelog for 1.1.0 is rather big, grouping it by engine, core, etc would help a lot. I already removed all the Python PRs to reduce the n

[GitHub] [iceberg] Fokko merged pull request #6396: Build: Bump certifi from 2022.9.24 to 2022.12.7 in /python

2022-12-09 Thread GitBox
Fokko merged PR #6396: URL: https://github.com/apache/iceberg/pull/6396 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] Fokko commented on a diff in pull request #6389: Python: Improve docs

2022-12-09 Thread GitBox
Fokko commented on code in PR #6389: URL: https://github.com/apache/iceberg/pull/6389#discussion_r1044353941 ## python/mkdocs/docs/index.md: ## @@ -17,507 +17,7 @@ # PyIceberg -Much of the python api conforms to the Java API. You can get more info about the java api [here

[GitHub] [iceberg] Fokko merged pull request #6384: Python: Improve `readme.md` "Getting started" by mentioning required pip version

2022-12-09 Thread GitBox
Fokko merged PR #6384: URL: https://github.com/apache/iceberg/pull/6384 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] Fokko commented on a diff in pull request #6389: Python: Improve docs

2022-12-09 Thread GitBox
Fokko commented on code in PR #6389: URL: https://github.com/apache/iceberg/pull/6389#discussion_r1044382982 ## python/mkdocs/docs/api.md: ## @@ -0,0 +1,336 @@ + + +# Python API + +PyIceberg is based around catalogs to load tables. First, we need to instantiate a catalog. Let's

[GitHub] [iceberg] Fokko commented on a diff in pull request #6389: Python: Improve docs

2022-12-09 Thread GitBox
Fokko commented on code in PR #6389: URL: https://github.com/apache/iceberg/pull/6389#discussion_r1044384157 ## python/mkdocs/docs/api.md: ## @@ -0,0 +1,336 @@ + + +# Python API + +PyIceberg is based around catalogs to load tables. First, we need to instantiate a catalog. Let's

[GitHub] [iceberg] Fokko commented on a diff in pull request #6389: Python: Improve docs

2022-12-09 Thread GitBox
Fokko commented on code in PR #6389: URL: https://github.com/apache/iceberg/pull/6389#discussion_r1044385272 ## python/mkdocs/docs/api.md: ## @@ -0,0 +1,336 @@ + + +# Python API + +PyIceberg is based around catalogs to load tables. First, we need to instantiate a catalog. Let's

[GitHub] [iceberg] Fokko commented on a diff in pull request #6389: Python: Improve docs

2022-12-09 Thread GitBox
Fokko commented on code in PR #6389: URL: https://github.com/apache/iceberg/pull/6389#discussion_r1044391915 ## python/mkdocs/docs/api.md: ## @@ -0,0 +1,336 @@ + + +# Python API + +PyIceberg is based around catalogs to load tables. First, we need to instantiate a catalog. Let's

[GitHub] [iceberg] Fokko commented on issue #3220: [Python] support iceberg hadoop catalog in python library

2022-12-09 Thread GitBox
Fokko commented on issue #3220: URL: https://github.com/apache/iceberg/issues/3220#issuecomment-1344240475 I'm seeing more and more people asking for this. The Hadoop catalog is a tricky implementation since it isn't supported my many object stores. Should we provide a `from_uri` method, to

[GitHub] [iceberg] Fokko commented on pull request #6389: Python: Improve docs

2022-12-09 Thread GitBox
Fokko commented on PR #6389: URL: https://github.com/apache/iceberg/pull/6389#issuecomment-1344241245 Thanks @rdblue @jun-he for the review, much appreciated 👍🏻 @djouallah Unfortunately this isn't supported currently, there is an outstanding issue: https://github.com/apache/iceberg/i

[GitHub] [iceberg] ajantha-bhat commented on issue #3220: [Python] support iceberg hadoop catalog in python library

2022-12-09 Thread GitBox
ajantha-bhat commented on issue #3220: URL: https://github.com/apache/iceberg/issues/3220#issuecomment-1344251262 IMO, Hadoop catalog is not a production recommended catalog (https://iceberg.apache.org/docs/latest/java-api-quickstart/#using-hadoop-tables) > Hadoop tables shouldn’t be

[GitHub] [iceberg] Fokko merged pull request #6389: Python: Improve docs

2022-12-09 Thread GitBox
Fokko merged PR #6389: URL: https://github.com/apache/iceberg/pull/6389 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apach

[GitHub] [iceberg] Fokko commented on issue #3220: [Python] support iceberg hadoop catalog in python library

2022-12-09 Thread GitBox
Fokko commented on issue #3220: URL: https://github.com/apache/iceberg/issues/3220#issuecomment-1344255902 @ajantha-bhat that's exactly my consideration as well. However, there are users that use for example ADLS, which has [support for atomic renames](https://learn.microsoft.com/en-us/azur

issues@iceberg.apache.org

2022-12-09 Thread GitBox
gaborkaszab commented on code in PR #6324: URL: https://github.com/apache/iceberg/pull/6324#discussion_r1044433283 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java: ## @@ -422,11 +425,21 @@ private Table newHmsTable(TableMetadata metadata) {

issues@iceberg.apache.org

2022-12-09 Thread GitBox
gaborkaszab commented on PR #6324: URL: https://github.com/apache/iceberg/pull/6324#issuecomment-1344302142 > Hey folks, comments from last round of review all taken and implemented. > > Specifically, on one comment: @gaborkaszab @szehon-ho I removed support for changing ownership for

[GitHub] [iceberg] pvary commented on issue #2301: Lock remains in HMS if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

2022-12-09 Thread GitBox
pvary commented on issue #2301: URL: https://github.com/apache/iceberg/issues/2301#issuecomment-1344352419 Are you doing the following: - Collect the new data files, - Refresh the table metadata, - Create a transaction and add the new files, - Commit the changes? -- This is an

[GitHub] [iceberg] stevenzwu merged pull request #6395: Flink: Fix minor compiler warning

2022-12-09 Thread GitBox
stevenzwu merged PR #6395: URL: https://github.com/apache/iceberg/pull/6395 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

[GitHub] [iceberg-docs] gaborkaszab commented on a diff in pull request #185: First version of the changelog

2022-12-09 Thread GitBox
gaborkaszab commented on code in PR #185: URL: https://github.com/apache/iceberg-docs/pull/185#discussion_r1044510801 ## landing-page/content/common/release-notes.md: ## @@ -70,8 +70,25 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ## 1.1.0

[GitHub] [iceberg] gaborkaszab commented on a diff in pull request #6293: Added FileIO Support for ORC Reader and Writers

2022-12-09 Thread GitBox
gaborkaszab commented on code in PR #6293: URL: https://github.com/apache/iceberg/pull/6293#discussion_r1044515746 ## orc/src/main/java/org/apache/iceberg/orc/ORC.java: ## @@ -85,10 +86,14 @@ import org.apache.orc.OrcFile.ReaderOptions; import org.apache.orc.Reader; import or

[GitHub] [iceberg] bryanck commented on a diff in pull request #6169: AWS,Core: Add S3 REST Signer client + REST Spec

2022-12-09 Thread GitBox
bryanck commented on code in PR #6169: URL: https://github.com/apache/iceberg/pull/6169#discussion_r1044555925 ## aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java: ## @@ -0,0 +1,321 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [iceberg] TuroczyX commented on issue #6347: [Docs]: improve ChangeLog

2022-12-09 Thread GitBox
TuroczyX commented on issue #6347: URL: https://github.com/apache/iceberg/issues/6347#issuecomment-1344452046 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6394: Flink: Port Support read options in flink source to 1.14 & 1.16

2022-12-09 Thread GitBox
stevenzwu commented on code in PR #6394: URL: https://github.com/apache/iceberg/pull/6394#discussion_r1044643611 ## flink/v1.15/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSource.java: ## @@ -112,8 +111,10 @@ public Builder project(TableSchema schema) { retur

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6394: Flink: Port Support read options in flink source to 1.14 & 1.16

2022-12-09 Thread GitBox
stevenzwu commented on code in PR #6394: URL: https://github.com/apache/iceberg/pull/6394#discussion_r1044644900 ## flink/v1.15/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSource.java: ## @@ -185,7 +186,7 @@ public Builder exposeLocality(boolean newExposeLocality) {

[GitHub] [iceberg] stevenzwu commented on a diff in pull request #6394: Flink: Port Support read options in flink source to 1.14 & 1.16

2022-12-09 Thread GitBox
stevenzwu commented on code in PR #6394: URL: https://github.com/apache/iceberg/pull/6394#discussion_r1044646329 ## flink/v1.15/flink/src/main/java/org/apache/iceberg/flink/source/IcebergTableSource.java: ## @@ -84,7 +84,7 @@ public IcebergTableSource( TableSchema schema,

[GitHub] [iceberg] stevenzwu commented on pull request #6394: Flink: Port Support read options in flink source to 1.14 & 1.16

2022-12-09 Thread GitBox
stevenzwu commented on PR #6394: URL: https://github.com/apache/iceberg/pull/6394#issuecomment-1344540034 @hililiwei after you reverted some changes in 1.15, please paste the `git diff` btw the modules here. e.g. https://github.com/apache/iceberg/pull/6363 -- This is an automated message

[GitHub] [iceberg] rdblue merged pull request #6386: Build: Bump Arrow from 7.0.0 to 10.0.1

2022-12-09 Thread GitBox
rdblue merged PR #6386: URL: https://github.com/apache/iceberg/pull/6386 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue commented on pull request #6386: Build: Bump Arrow from 7.0.0 to 10.0.1

2022-12-09 Thread GitBox
rdblue commented on PR #6386: URL: https://github.com/apache/iceberg/pull/6386#issuecomment-1344562650 Thanks, @nastra! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

[GitHub] [iceberg] rdblue commented on a diff in pull request #6074: API,Core: SnapshotManager to be created through Transaction

2022-12-09 Thread GitBox
rdblue commented on code in PR #6074: URL: https://github.com/apache/iceberg/pull/6074#discussion_r1044670805 ## .palantir/revapi.yml: ## @@ -43,9 +49,6 @@ acceptedBreaks: - code: "java.method.removed" old: "method org.apache.iceberg.RowDelta org.apache.iceberg.RowD

[GitHub] [iceberg] rdblue commented on a diff in pull request #6074: API,Core: SnapshotManager to be created through Transaction

2022-12-09 Thread GitBox
rdblue commented on code in PR #6074: URL: https://github.com/apache/iceberg/pull/6074#discussion_r1044672058 ## core/src/main/java/org/apache/iceberg/SnapshotManager.java: ## @@ -30,6 +31,17 @@ public class SnapshotManager implements ManageSnapshots { ops.current() !=

[GitHub] [iceberg] rdblue commented on a diff in pull request #6348: Python: Update license-checker

2022-12-09 Thread GitBox
rdblue commented on code in PR #6348: URL: https://github.com/apache/iceberg/pull/6348#discussion_r1044686299 ## python/dev/.rat-excludes: ## @@ -0,0 +1,2 @@ +.rat-excludes Review Comment: We can keep the two separate but use the same script right? I'd prefer that since it

[GitHub] [iceberg] rdblue merged pull request #6341: Docs: Update PyIceberg release guide

2022-12-09 Thread GitBox
rdblue merged PR #6341: URL: https://github.com/apache/iceberg/pull/6341 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue merged pull request #6329: Python: Add support for 3.11

2022-12-09 Thread GitBox
rdblue merged PR #6329: URL: https://github.com/apache/iceberg/pull/6329 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

[GitHub] [iceberg] rdblue closed issue #6124: Support Python 3.11

2022-12-09 Thread GitBox
rdblue closed issue #6124: Support Python 3.11 URL: https://github.com/apache/iceberg/issues/6124 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues

[GitHub] [iceberg] RussellSpitzer opened a new issue, #6397: Python Instructions currently do not work for testing

2022-12-09 Thread GitBox
RussellSpitzer opened a new issue, #6397: URL: https://github.com/apache/iceberg/issues/6397 ### Apache Iceberg version main (development) ### Query engine Other ### Please describe the bug 🐞 The instructions listed in the README.md under testing ```b

[GitHub] [iceberg] rdblue commented on issue #3220: [Python] support iceberg hadoop catalog in python library

2022-12-09 Thread GitBox
rdblue commented on issue #3220: URL: https://github.com/apache/iceberg/issues/3220#issuecomment-1344611016 For what it's worth, I think that **the biggest mistake I made with Iceberg was introducing the "Hadoop" tables** that rely on atomic rename. These tables have a lot of problems

[GitHub] [iceberg] rubenvdg closed issue #6383: Docs: Improve "Getting started" by mentioning required pip version

2022-12-09 Thread GitBox
rubenvdg closed issue #6383: Docs: Improve "Getting started" by mentioning required pip version URL: https://github.com/apache/iceberg/issues/6383 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [iceberg] rdblue commented on a diff in pull request #3231: GCM encryption stream

2022-12-09 Thread GitBox
rdblue commented on code in PR #3231: URL: https://github.com/apache/iceberg/pull/3231#discussion_r1044706354 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputStream.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

[GitHub] [iceberg] rdblue commented on a diff in pull request #3231: GCM encryption stream

2022-12-09 Thread GitBox
rdblue commented on code in PR #3231: URL: https://github.com/apache/iceberg/pull/3231#discussion_r1044708947 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputStream.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

[GitHub] [iceberg] rdblue commented on a diff in pull request #3231: GCM encryption stream

2022-12-09 Thread GitBox
rdblue commented on code in PR #3231: URL: https://github.com/apache/iceberg/pull/3231#discussion_r1044708947 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputStream.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

[GitHub] [iceberg] rdblue commented on a diff in pull request #3231: GCM encryption stream

2022-12-09 Thread GitBox
rdblue commented on code in PR #3231: URL: https://github.com/apache/iceberg/pull/3231#discussion_r1044720784 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputStream.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

[GitHub] [iceberg] rdblue commented on a diff in pull request #3231: GCM encryption stream

2022-12-09 Thread GitBox
rdblue commented on code in PR #3231: URL: https://github.com/apache/iceberg/pull/3231#discussion_r1044723489 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputStream.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

[GitHub] [iceberg] rdblue commented on a diff in pull request #3231: GCM encryption stream

2022-12-09 Thread GitBox
rdblue commented on code in PR #3231: URL: https://github.com/apache/iceberg/pull/3231#discussion_r1044724923 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputStream.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

[GitHub] [iceberg] rdblue commented on a diff in pull request #3231: GCM encryption stream

2022-12-09 Thread GitBox
rdblue commented on code in PR #3231: URL: https://github.com/apache/iceberg/pull/3231#discussion_r1044725178 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputStream.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

[GitHub] [iceberg] rdblue commented on a diff in pull request #3231: GCM encryption stream

2022-12-09 Thread GitBox
rdblue commented on code in PR #3231: URL: https://github.com/apache/iceberg/pull/3231#discussion_r1044726532 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputStream.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

[GitHub] [iceberg] rdblue commented on a diff in pull request #3231: GCM encryption stream

2022-12-09 Thread GitBox
rdblue commented on code in PR #3231: URL: https://github.com/apache/iceberg/pull/3231#discussion_r1044728554 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputStream.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

[GitHub] [iceberg] rdblue commented on a diff in pull request #3231: GCM encryption stream

2022-12-09 Thread GitBox
rdblue commented on code in PR #3231: URL: https://github.com/apache/iceberg/pull/3231#discussion_r1044743460 ## core/src/main/java/org/apache/iceberg/encryption/AesGcmInputStream.java: ## @@ -0,0 +1,218 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

[GitHub] [iceberg] Fokko commented on a diff in pull request #6348: Python: Update license-checker

2022-12-09 Thread GitBox
Fokko commented on code in PR #6348: URL: https://github.com/apache/iceberg/pull/6348#discussion_r1044764109 ## python/dev/.rat-excludes: ## @@ -0,0 +1,2 @@ +.rat-excludes Review Comment: This will make the building more complicated. Before doing a `poetry build`, we need t

[GitHub] [iceberg] rdblue commented on a diff in pull request #3231: GCM encryption stream

2022-12-09 Thread GitBox
rdblue commented on code in PR #3231: URL: https://github.com/apache/iceberg/pull/3231#discussion_r1044792391 ## core/src/main/java/org/apache/iceberg/encryption/Ciphers.java: ## @@ -96,33 +110,45 @@ public AesGcmDecryptor(byte[] keyBytes) { } public byte[] decrypt(b

[GitHub] [iceberg] xinbinhuang commented on pull request #3059: Returns isUnpartitioned=true for VoidTransform on all fields

2022-12-09 Thread GitBox
xinbinhuang commented on PR #3059: URL: https://github.com/apache/iceberg/pull/3059#issuecomment-1344748524 (@RussellSpitzer sorry didn't see the last message from you) @RussellSpitzer @rdblue Just rebased. PTAL -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #3059: Returns isUnpartitioned=true for VoidTransform on all fields

2022-12-09 Thread GitBox
RussellSpitzer commented on code in PR #3059: URL: https://github.com/apache/iceberg/pull/3059#discussion_r1044822325 ## core/src/test/java/org/apache/iceberg/TestPartitionSpecInfo.java: ## @@ -65,11 +65,22 @@ public void cleanupTables() { TestTables.clearTables(); } +

[GitHub] [iceberg] RussellSpitzer commented on pull request #3059: Returns isUnpartitioned=true for VoidTransform on all fields

2022-12-09 Thread GitBox
RussellSpitzer commented on PR #3059: URL: https://github.com/apache/iceberg/pull/3059#issuecomment-1344753963 @xinbinhuang Looks good to me, once tests pass I think we are good to go. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
RussellSpitzer commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044825312 ## core/src/main/java/org/apache/iceberg/util/SnapshotUtil.java: ## @@ -150,7 +150,7 @@ public static Iterable ancestorsOf(long snapshotId, Function

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
RussellSpitzer commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044826932 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestChangelogTable.java: ## @@ -137,6 +138,64 @@ public void testOverwrites() {

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
RussellSpitzer commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044836296 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestChangelogTable.java: ## @@ -137,6 +138,64 @@ public void testOverwrites() {

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
RussellSpitzer commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044841503 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestChangelogTable.java: ## @@ -137,6 +138,64 @@ public void testOverwrites() {

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
RussellSpitzer commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044857145 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -308,6 +339,17 @@ public Scan buildChangelogScan() { return n

[GitHub] [iceberg] flyrain commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
flyrain commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044874538 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestChangelogTable.java: ## @@ -137,6 +138,64 @@ public void testOverwrites() {

[GitHub] [iceberg] Fokko opened a new pull request, #6398: Python: Integration tests

2022-12-09 Thread GitBox
Fokko opened a new pull request, #6398: URL: https://github.com/apache/iceberg/pull/6398 This is the first version of a framework to read Iceberg tables, produced by Spark, using PyIceberg. This makes it easier to run end-to-end tests and also validate the behavior of PyArrow and DuckDB.

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

2022-12-09 Thread GitBox
szehon-ho commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1044799711 ## api/src/main/java/org/apache/iceberg/ChangelogOperation.java: ## @@ -21,5 +21,7 @@ /** An enum representing possible operations in a changelog. */ public enum Ch

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

2022-12-09 Thread GitBox
szehon-ho commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1044799711 ## api/src/main/java/org/apache/iceberg/ChangelogOperation.java: ## @@ -21,5 +21,7 @@ /** An enum representing possible operations in a changelog. */ public enum Ch

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

2022-12-09 Thread GitBox
szehon-ho commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1044855421 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

2022-12-09 Thread GitBox
szehon-ho commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1044848715 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

2022-12-09 Thread GitBox
szehon-ho commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1044889725 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

2022-12-09 Thread GitBox
RussellSpitzer commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1044893113 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [iceberg-docs] Fokko commented on a diff in pull request #185: First version of the changelog

2022-12-09 Thread GitBox
Fokko commented on code in PR #185: URL: https://github.com/apache/iceberg-docs/pull/185#discussion_r1044893253 ## landing-page/content/common/release-notes.md: ## @@ -70,8 +70,25 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ## 1.1.0 relea

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

2022-12-09 Thread GitBox
RussellSpitzer commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1044893660 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [iceberg-docs] Fokko commented on a diff in pull request #185: First version of the changelog

2022-12-09 Thread GitBox
Fokko commented on code in PR #185: URL: https://github.com/apache/iceberg-docs/pull/185#discussion_r1044893793 ## landing-page/content/common/release-notes.md: ## @@ -70,8 +70,25 @@ To add a dependency on Iceberg in Maven, add the following to your `pom.xml`: ## 1.1.0 relea

[GitHub] [iceberg] flyrain commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
flyrain commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044896124 ## core/src/main/java/org/apache/iceberg/util/SnapshotUtil.java: ## @@ -150,7 +150,7 @@ public static Iterable ancestorsOf(long snapshotId, Function

[GitHub] [iceberg] rdblue commented on a diff in pull request #6072: Core: Add scan report for incremental Table scans

2022-12-09 Thread GitBox
rdblue commented on code in PR #6072: URL: https://github.com/apache/iceberg/pull/6072#discussion_r1044896209 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -118,6 +119,10 @@ public static Set getProjectedIds(Schema schema) { return ImmutableSet.copyOf(g

[GitHub] [iceberg] rdblue commented on a diff in pull request #6072: Core: Add scan report for incremental Table scans

2022-12-09 Thread GitBox
rdblue commented on code in PR #6072: URL: https://github.com/apache/iceberg/pull/6072#discussion_r1044896209 ## api/src/main/java/org/apache/iceberg/types/TypeUtil.java: ## @@ -118,6 +119,10 @@ public static Set getProjectedIds(Schema schema) { return ImmutableSet.copyOf(g

[GitHub] [iceberg] RussellSpitzer merged pull request #3059: Returns isUnpartitioned=true for VoidTransform on all fields

2022-12-09 Thread GitBox
RussellSpitzer merged PR #3059: URL: https://github.com/apache/iceberg/pull/3059 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceb

[GitHub] [iceberg] RussellSpitzer closed issue #3014: PartitionSpec isUnpartitioned returns true for tables which previously had Partitions but no longer do

2022-12-09 Thread GitBox
RussellSpitzer closed issue #3014: PartitionSpec isUnpartitioned returns true for tables which previously had Partitions but no longer do URL: https://github.com/apache/iceberg/issues/3014 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [iceberg] flyrain commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
flyrain commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044897001 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkReadOptions.java: ## @@ -32,6 +32,12 @@ private SparkReadOptions() {} // End snapshot ID used in incr

[GitHub] [iceberg] rdblue commented on a diff in pull request #6072: Core: Add scan report for incremental Table scans

2022-12-09 Thread GitBox
rdblue commented on code in PR #6072: URL: https://github.com/apache/iceberg/pull/6072#discussion_r1044897257 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -165,6 +170,13 @@ public CloseableIterable apply( context.residual

[GitHub] [iceberg] flyrain commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
flyrain commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044897305 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -285,6 +286,36 @@ public Scan buildChangelogScan() { Long startSna

[GitHub] [iceberg] flyrain commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
flyrain commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044897410 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -285,6 +286,36 @@ public Scan buildChangelogScan() { Long startSna

[GitHub] [iceberg] flyrain commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
flyrain commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044897546 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestChangelogTable.java: ## @@ -137,6 +138,64 @@ public void testOverwrites() {

[GitHub] [iceberg] RussellSpitzer commented on pull request #3059: Returns isUnpartitioned=true for VoidTransform on all fields

2022-12-09 Thread GitBox
RussellSpitzer commented on PR #3059: URL: https://github.com/apache/iceberg/pull/3059#issuecomment-1344862752 My commit title was inverted, mia culpa. For anyone looking this up in the future I meant that "all void transforms should be false" -- This is an automated message from the Apac

[GitHub] [iceberg] Fokko commented on issue #5901: pip install pyiceberg on windows require C++ to be installed

2022-12-09 Thread GitBox
Fokko commented on issue #5901: URL: https://github.com/apache/iceberg/issues/5901#issuecomment-1344862830 Awesome, thanks for letting us know! @djouallah -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [iceberg] flyrain commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
flyrain commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044899026 ## spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestChangelogTable.java: ## @@ -137,6 +138,64 @@ public void testOverwrites() {

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

2022-12-09 Thread GitBox
RussellSpitzer commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r104490 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #6344: Spark 3.3: Introduce the changelog iterator

2022-12-09 Thread GitBox
RussellSpitzer commented on code in PR #6344: URL: https://github.com/apache/iceberg/pull/6344#discussion_r1044900565 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/ChangelogIterator.java: ## @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [iceberg] flyrain commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
flyrain commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044901322 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -308,6 +339,17 @@ public Scan buildChangelogScan() { return new Spar

[GitHub] [iceberg] szehon-ho merged pull request #6354: Spark: Check fileIO instead of reading location when determining locality enabled

2022-12-09 Thread GitBox
szehon-ho merged PR #6354: URL: https://github.com/apache/iceberg/pull/6354 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

[GitHub] [iceberg] szehon-ho commented on pull request #6354: Spark: Check fileIO instead of reading location when determining locality enabled

2022-12-09 Thread GitBox
szehon-ho commented on PR #6354: URL: https://github.com/apache/iceberg/pull/6354#issuecomment-1344869084 Merged, thanks @amogh-jahagirdar for the change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [iceberg] flyrain commented on pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
flyrain commented on PR #6350: URL: https://github.com/apache/iceberg/pull/6350#issuecomment-1344870088 Hi @szehon-ho, @RussellSpitzer, @hililiwei thanks for the review! Resolved your comments. Take another look? -- This is an automated message from the Apache Git Service. To respond to t

[GitHub] [iceberg] flyrain commented on a diff in pull request #6350: Query changelog table with a timestamp range

2022-12-09 Thread GitBox
flyrain commented on code in PR #6350: URL: https://github.com/apache/iceberg/pull/6350#discussion_r1044906141 ## spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java: ## @@ -308,6 +339,17 @@ public Scan buildChangelogScan() { return new Spar

  1   2   >