[PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.21.2 to 1.23.1 [iceberg-go]
dependabot[bot] opened a new pull request, #39: URL: https://github.com/apache/iceberg-go/pull/39 Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.21.2 to 1.23.1. Commits https://github.com/aws/aws-sdk-go-v2/commit/b3e07aa0a0dd26ec46095c28ce65301da2e78dba";>b3e07aa Release 2023-11-20 https://github.com/aws/aws-sdk-go-v2/commit/2fc1c0dab14dff82fc2c2465cb7c40a4157196a0";>2fc1c0d Regenerated Clients https://github.com/aws/aws-sdk-go-v2/commit/1a6bd026243230178e775059d27038c0ced194ea";>1a6bd02 Update endpoints model https://github.com/aws/aws-sdk-go-v2/commit/ac2c871bff4a9fbb6284f1dcc061fdcabea346f3";>ac2c871 Update API model https://github.com/aws/aws-sdk-go-v2/commit/03c6858107f8c4c85637c18e1b643e33ca4575c0";>03c6858 deprecate v4.SignHTTPRequestMiddleware (https://redirect.github.com/aws/aws-sdk-go-v2/issues/2375";>#2375) https://github.com/aws/aws-sdk-go-v2/commit/0be05fa6cc06d8e81eda10cbf8413030754c4df6";>0be05fa Release 2023-11-17 https://github.com/aws/aws-sdk-go-v2/commit/d0c9d422756e7eb30b9937468e659b140fc5e578";>d0c9d42 Regenerated Clients https://github.com/aws/aws-sdk-go-v2/commit/e2ede4019d693a352b0ccbb0b87dd74c1fb4990e";>e2ede40 Update endpoints model https://github.com/aws/aws-sdk-go-v2/commit/7720f87d0ec2f3609e2912c864b6f9ae550e0792";>7720f87 Update API model https://github.com/aws/aws-sdk-go-v2/commit/4bd06b93f8627928c17604383bd8639f2cb23739";>4bd06b9 Merge customizations for service s3 Additional commits viewable in https://github.com/aws/aws-sdk-go-v2/compare/v1.21.2...v1.23.1";>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.21.2 to 1.23.0 [iceberg-go]
dependabot[bot] closed pull request #36: build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.21.2 to 1.23.0 URL: https://github.com/apache/iceberg-go/pull/36 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.21.2 to 1.23.0 [iceberg-go]
dependabot[bot] commented on PR #36: URL: https://github.com/apache/iceberg-go/pull/36#issuecomment-1826720525 Superseded by #39. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
[PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.19.1 to 1.25.5 [iceberg-go]
dependabot[bot] opened a new pull request, #40: URL: https://github.com/apache/iceberg-go/pull/40 Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.19.1 to 1.25.5. Commits https://github.com/aws/aws-sdk-go-v2/commit/fa3ee1a83b9edac9669159650622bcfe3003b44c";>fa3ee1a Release 2023-11-21 https://github.com/aws/aws-sdk-go-v2/commit/6e0c553b49b1fd84753d3ea440be93011d36cf40";>6e0c553 Regenerated Clients https://github.com/aws/aws-sdk-go-v2/commit/32fa00b32e99f9004aadc5750d508a9393d03d57";>32fa00b Update endpoints model https://github.com/aws/aws-sdk-go-v2/commit/72eae5c284e11b1f167fe17282ae1ee821a11659";>72eae5c Update API model https://github.com/aws/aws-sdk-go-v2/commit/1c69d08daccf94d9114d2463adc02e59e85b4deb";>1c69d08 fix: don't expect response to be json in endpointcreds provider (https://redirect.github.com/aws/aws-sdk-go-v2/issues/2381";>#2381) https://github.com/aws/aws-sdk-go-v2/commit/3bd97c063d962a34ca496720a3ce00ef4affe5fd";>3bd97c0 fix: don't send 0 max items for object version and multipart upload paginator... https://github.com/aws/aws-sdk-go-v2/commit/b3e07aa0a0dd26ec46095c28ce65301da2e78dba";>b3e07aa Release 2023-11-20 https://github.com/aws/aws-sdk-go-v2/commit/2fc1c0dab14dff82fc2c2465cb7c40a4157196a0";>2fc1c0d Regenerated Clients https://github.com/aws/aws-sdk-go-v2/commit/1a6bd026243230178e775059d27038c0ced194ea";>1a6bd02 Update endpoints model https://github.com/aws/aws-sdk-go-v2/commit/ac2c871bff4a9fbb6284f1dcc061fdcabea346f3";>ac2c871 Update API model Additional commits viewable in https://github.com/aws/aws-sdk-go-v2/compare/v1.19.1...config/v1.25.5";>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.19.1 to 1.25.3 [iceberg-go]
dependabot[bot] commented on PR #38: URL: https://github.com/apache/iceberg-go/pull/38#issuecomment-1826720682 Superseded by #40. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.19.1 to 1.25.3 [iceberg-go]
dependabot[bot] closed pull request #38: build(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.19.1 to 1.25.3 URL: https://github.com/apache/iceberg-go/pull/38 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] build(deps): bump github.com/wolfeidau/s3iofs from 1.3.0 to 1.3.1 [iceberg-go]
dependabot[bot] commented on PR #31: URL: https://github.com/apache/iceberg-go/pull/31#issuecomment-1826720860 Superseded by #41. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
[PR] build(deps): bump github.com/wolfeidau/s3iofs from 1.3.0 to 1.4.0 [iceberg-go]
dependabot[bot] opened a new pull request, #41: URL: https://github.com/apache/iceberg-go/pull/41 Bumps [github.com/wolfeidau/s3iofs](https://github.com/wolfeidau/s3iofs) from 1.3.0 to 1.4.0. Release notes Sourced from https://github.com/wolfeidau/s3iofs/releases";>github.com/wolfeidau/s3iofs's releases. v1.4.0 What's Changed chore(deps): bump github.com/docker/docker from 24.0.6+incompatible to 24.0.7+incompatible in /integration by https://github.com/dependabot";>@dependabot in https://redirect.github.com/wolfeidau/s3iofs/pull/21";>wolfeidau/s3iofs#21 chore(deps): upgrade aws sdkv2 for https://redirect.github.com/aws/aws-sdk-go-v2/issues/2";>aws/aws-sdk-go-v22 by https://github.com/wolfeidau";>@wolfeidau in https://redirect.github.com/wolfeidau/s3iofs/pull/22";>wolfeidau/s3iofs#22 Full Changelog: https://github.com/wolfeidau/s3iofs/compare/v1.3.1...v1.4.0";>https://github.com/wolfeidau/s3iofs/compare/v1.3.1...v1.4.0 v1.3.1 What's Changed docs(README): add some badges with a godoc link by https://github.com/wolfeidau";>@wolfeidau in https://redirect.github.com/wolfeidau/s3iofs/pull/14";>wolfeidau/s3iofs#14 feat(testing): increase integration test coverage :rocket: by https://github.com/wolfeidau";>@wolfeidau in https://redirect.github.com/wolfeidau/s3iofs/pull/15";>wolfeidau/s3iofs#15 feat(tests): added flags for vscode to ensure integration test coverage works by https://github.com/wolfeidau";>@wolfeidau in https://redirect.github.com/wolfeidau/s3iofs/pull/16";>wolfeidau/s3iofs#16 chore(deps): upgrade go deps by https://github.com/wolfeidau";>@wolfeidau in https://redirect.github.com/wolfeidau/s3iofs/pull/19";>wolfeidau/s3iofs#19 chore(deps): upgrade go deps for integration tests by https://github.com/wolfeidau";>@wolfeidau in https://redirect.github.com/wolfeidau/s3iofs/pull/20";>wolfeidau/s3iofs#20 Full Changelog: https://github.com/wolfeidau/s3iofs/compare/v1.3.0...v1.3.1";>https://github.com/wolfeidau/s3iofs/compare/v1.3.0...v1.3.1 Commits https://github.com/wolfeidau/s3iofs/commit/dd935dd4da613eeef2bb465c46ce3b0df5bbaa92";>dd935dd Merge pull request https://redirect.github.com/wolfeidau/s3iofs/issues/22";>#22 from wolfeidau/fix_upgrade_aws_sdkv2 https://github.com/wolfeidau/s3iofs/commit/44d4dcf1d92c01b43a4a9719037be231556f8295";>44d4dcf chore(deps): upgrade aws sdkv2 for https://redirect.github.com/aws/aws-sdk-go-v2/issues/2";>aws/aws-sdk-go-v22 https://github.com/wolfeidau/s3iofs/commit/be25b7cccb36def0431eee83c1844706ea8feaa7";>be25b7c Merge pull request https://redirect.github.com/wolfeidau/s3iofs/issues/21";>#21 from wolfeidau/dependabot/go_modules/integration/githu... https://github.com/wolfeidau/s3iofs/commit/c726f39e0fd8896bd9669820cfd0bd19d8d8a5f9";>c726f39 chore(deps): bump github.com/docker/docker in /integration https://github.com/wolfeidau/s3iofs/commit/710788272cd775c490622c9fd2d56a25ea138929";>7107882 Merge pull request https://redirect.github.com/wolfeidau/s3iofs/issues/20";>#20 from wolfeidau/chore_upgrade_integration_deps https://github.com/wolfeidau/s3iofs/commit/8e14816297b4761912d1f65d7b25ffa5145d1a41";>8e14816 chore(deps): upgrade go deps for integration tests https://github.com/wolfeidau/s3iofs/commit/87378762a59e2b2ec85822e8e217a4322771db39";>8737876 Merge pull request https://redirect.github.com/wolfeidau/s3iofs/issues/19";>#19 from wolfeidau/chore_oct_dep_upgrades https://github.com/wolfeidau/s3iofs/commit/5bcee15b28710992fea999ddacd931e206eccef2";>5bcee15 chore(deps): upgrade go deps https://github.com/wolfeidau/s3iofs/commit/ba8909f07876d88ae05ae3cec4756736bf185371";>ba8909f Merge pull request https://redirect.github.com/wolfeidau/s3iofs/issues/16";>#16 from wolfeidau/feat_vscode_test_coverage https://github.com/wolfeidau/s3iofs/commit/385abc4f78bff56a39ff673baaf8f306b19cfd40";>385abc4 feat(tests): added flags for vscode to ensure integration test coverage works Additional commits viewable in https://github.com/wolfeidau/s3iofs/compare/v1.3.0...v1.4.0";>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any ed
Re: [PR] build(deps): bump github.com/wolfeidau/s3iofs from 1.3.0 to 1.3.1 [iceberg-go]
dependabot[bot] closed pull request #31: build(deps): bump github.com/wolfeidau/s3iofs from 1.3.0 to 1.3.1 URL: https://github.com/apache/iceberg-go/pull/31 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] Flink: Read parquet BINARY column as String for expected [iceberg]
fengjiajie commented on PR #8808: URL: https://github.com/apache/iceberg/pull/8808#issuecomment-1826731977 > You can only guarantee this is safe for your data, for any other user this could be unsafe. That’s the underlying issue with this PR, we are essentially allowing a cast binary as string.Sent from my iPhoneOn Nov 24, 2023, at 4:47 AM, fengjiajie ***@***.***> wrote: I'm also a little nervous about this change, how are we guaranteed that the binary is parsable as UTF8 bytes? Seems like we should just be fixing the type annotations rather than changing our readers to read files that have been written incorrectly? @RussellSpitzer Hi, can you please tell if this issue can be moved forward? We have a lot of hive tables that contain such parquet files and we are trying to convert these hive tables into iceberg tables, this process of parquet files cannot be rewritten (because of the large number of history files). We can guarantee that it could be parsed in UTF-8 because the data was originally defined as a string in hive. If it wasn't a string before, there's no reason defining it as a string when defining the iceberg table would make it fail to parse. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***> @RussellSpitzer Thanks for the reply, but I still don't get it. * I don't quite understand why this is 'unsafe' for any other user? * Also this conversion is only possible if the user defines the iceberg column as a string. A user defining a column as string means that the user wants to use the value of the column as a string (string is required by the iceberg specification to be UTF-8 encoded, and the library decodes it as UTF-8 according to the specification), otherwise, if the user only wants the column to be used as a binary, the user should define the iceberg column as a binary type instead of a string type, and there would be no conversion. Anyway, the conversion is based on the fact that the user defines the column as string and wants to use it as a string. If you think there is an inappropriate scenario, could you give an example? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
[PR] chore(deps): Update uuid requirement from ~1.5.0 to ~1.6.1 [iceberg-rust]
dependabot[bot] opened a new pull request, #107: URL: https://github.com/apache/iceberg-rust/pull/107 Updates the requirements on [uuid](https://github.com/uuid-rs/uuid) to permit the latest version. Release notes Sourced from https://github.com/uuid-rs/uuid/releases";>uuid's releases. 1.5.0 What's Changed Add impl From for String under the std feature flag by https://github.com/brahms116";>@brahms116 in https://redirect.github.com/uuid-rs/uuid/pull/700";>uuid-rs/uuid#700 Remove dead link to templates by https://github.com/KodrAus";>@KodrAus in https://redirect.github.com/uuid-rs/uuid/pull/704";>uuid-rs/uuid#704 make ClockSequence wrap correctly by https://github.com/fef1312";>@fef1312 in https://redirect.github.com/uuid-rs/uuid/pull/705";>uuid-rs/uuid#705 Track MSRV in Cargo.toml by https://github.com/KodrAus";>@KodrAus in https://redirect.github.com/uuid-rs/uuid/pull/706";>uuid-rs/uuid#706 Support converting between Uuid and vec by https://github.com/KodrAus";>@KodrAus in https://redirect.github.com/uuid-rs/uuid/pull/703";>uuid-rs/uuid#703 Replace MIPS with Miri and add clippy to CI by https://github.com/KodrAus";>@KodrAus in https://redirect.github.com/uuid-rs/uuid/pull/712";>uuid-rs/uuid#712 Added bytemuck support by https://github.com/John-Toohey";>@John-Toohey in https://redirect.github.com/uuid-rs/uuid/pull/711";>uuid-rs/uuid#711 Prepare for 1.5.0 release by https://github.com/KodrAus";>@KodrAus in https://redirect.github.com/uuid-rs/uuid/pull/713";>uuid-rs/uuid#713 New Contributors https://github.com/brahms116";>@brahms116 made their first contribution in https://redirect.github.com/uuid-rs/uuid/pull/700";>uuid-rs/uuid#700 https://github.com/fef1312";>@fef1312 made their first contribution in https://redirect.github.com/uuid-rs/uuid/pull/705";>uuid-rs/uuid#705 https://github.com/John-Toohey";>@John-Toohey made their first contribution in https://redirect.github.com/uuid-rs/uuid/pull/711";>uuid-rs/uuid#711 Full Changelog: https://github.com/uuid-rs/uuid/compare/1.4.1...1.5.0";>https://github.com/uuid-rs/uuid/compare/1.4.1...1.5.0 Commits See full diff in https://github.com/uuid-rs/uuid/compare/1.5.0...1.5.0";>compare view Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] HIVE_METASTORE_ERROR: Table storage descriptor is missing SerDe info - when query a view using an Iceberg table on Athena [iceberg]
yakovsushenok commented on issue #4549: URL: https://github.com/apache/iceberg/issues/4549#issuecomment-1826790324 I used iceberg table format v2 and when I was querying with Athena engine 2, I was getting this error. When I changed to Athena engine 3 this error went away. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] inputFormat, outputFormat, and serialization.lib not being set with AWS Glue 4.0 and Iceberg while Create table [iceberg]
yakovsushenok commented on issue #6783: URL: https://github.com/apache/iceberg/issues/6783#issuecomment-1826790765 I used iceberg table format v2 and when I was querying with Athena engine 2, I was getting HIVE_METASTORE_ERROR: Table storage descriptor is missing SerDe info. When I changed to Athena engine 3 this error went away. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] Replace black by Ruff Formatter [iceberg-python]
hussein-awala commented on PR #127: URL: https://github.com/apache/iceberg-python/pull/127#issuecomment-1826859402 > Looks fine overall, but it seems like too many changes with string normalization. Why force string normalization? That's going to cause a ton of pull requests to fail formatting validation. It is not supported yet; we can wait for https://github.com/astral-sh/ruff/pull/8822 before merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] Replace black by Ruff Formatter [iceberg-python]
hussein-awala commented on code in PR #127: URL: https://github.com/apache/iceberg-python/pull/127#discussion_r1405455829 ## .pre-commit-config.yaml: ## @@ -29,15 +29,11 @@ repos: - id: check-ast - repo: https://github.com/astral-sh/ruff-pre-commit # Ruff version (Used for linting) -rev: v0.0.291 Review Comment: yes it was introduced in https://github.com/astral-sh/ruff/releases/tag/v0.0.290, but I used the newer version to get some patches. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] A good Example to leverage glue(Aws integration with spark) is missing right now [iceberg]
github-actions[bot] commented on issue #7735: URL: https://github.com/apache/iceberg/issues/7735#issuecomment-1826951487 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] Parquet : spark config "spark.sql.parquet.writeLegacyFormat = true " is useless [iceberg]
github-actions[bot] commented on issue #7706: URL: https://github.com/apache/iceberg/issues/7706#issuecomment-1826951508 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] Arrow reader fails to filter results [iceberg]
github-actions[bot] commented on issue #7556: URL: https://github.com/apache/iceberg/issues/7556#issuecomment-1826951581 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] Delete across all snapshots [iceberg]
github-actions[bot] commented on issue #7622: URL: https://github.com/apache/iceberg/issues/7622#issuecomment-1826951542 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] Arrow reader fails to filter results [iceberg]
github-actions[bot] closed issue #7556: Arrow reader fails to filter results URL: https://github.com/apache/iceberg/issues/7556 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] Quickstart should give an example of a REST catalog enablement in the spark iceberg docker compose setup [iceberg]
github-actions[bot] commented on issue #7615: URL: https://github.com/apache/iceberg/issues/7615#issuecomment-1826951552 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] Delete across all snapshots [iceberg]
github-actions[bot] closed issue #7622: Delete across all snapshots URL: https://github.com/apache/iceberg/issues/7622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] Nessie: Support views for NessieCatalog [iceberg]
ajantha-bhat commented on code in PR #8909: URL: https://github.com/apache/iceberg/pull/8909#discussion_r1405528927 ## nessie/src/main/java/org/apache/iceberg/nessie/NessieUtil.java: ## @@ -165,4 +180,95 @@ public static TableMetadata updateTableMetadataWithNessieSpecificProperties( return builder.discardChanges().build(); } + + public static ViewMetadata loadViewMetadata( Review Comment: This is similar to `updateTableMetadataWithNessieSpecificProperties` present in this file. Trino will also use this method. Trino will not use `NessieViewOperations` as it has to have its own class. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] Nessie: Support views for NessieCatalog [iceberg]
ajantha-bhat commented on code in PR #8909:
URL: https://github.com/apache/iceberg/pull/8909#discussion_r1405536815
##
nessie/src/main/java/org/apache/iceberg/nessie/NessieIcebergClient.java:
##
@@ -378,27 +403,71 @@ public void renameTable(TableIdentifier from,
TableIdentifier to) {
// behavior. So better be safe than sorry.
}
+ private static void validateToContentForRename(
+ TableIdentifier from, TableIdentifier to, IcebergContent
existingToContent) {
+if (existingToContent != null) {
+ if (existingToContent.getType() == Content.Type.ICEBERG_VIEW) {
+throw new AlreadyExistsException("Cannot rename %s to %s. View already
exists", from, to);
+ } else if (existingToContent.getType() == Content.Type.ICEBERG_TABLE) {
+throw new AlreadyExistsException("Cannot rename %s to %s. Table
already exists", from, to);
+ } else {
+throw new AlreadyExistsException(
+"Cannot rename %s to %s. Another content with same name already
exists", from, to);
+ }
+}
+ }
+
+ private static void validateFromContentForRename(
+ TableIdentifier from, Content.Type type, IcebergContent
existingFromContent) {
+if (existingFromContent == null) {
+ if (type == Content.Type.ICEBERG_VIEW) {
+throw new NoSuchViewException("View does not exist: %s", from);
+ } else if (type == Content.Type.ICEBERG_TABLE) {
+throw new NoSuchTableException("Table does not exist: %s", from);
+ } else {
+throw new UnsupportedOperationException("Cannot perform rename for
content type: " + type);
+ }
+} else if (existingFromContent.getType() != type) {
+ throw new UnsupportedOperationException(
+ String.format("content type of from identifier %s should be of %s",
from, type));
+}
+ }
+
public boolean dropTable(TableIdentifier identifier, boolean purge) {
+return dropContent(identifier, purge, Content.Type.ICEBERG_TABLE);
+ }
+
+ public boolean dropView(TableIdentifier identifier, boolean purge) {
+return dropContent(identifier, purge, Content.Type.ICEBERG_VIEW);
+ }
+
+ private boolean dropContent(TableIdentifier identifier, boolean purge,
Content.Type type) {
getRef().checkMutable();
-IcebergTable existingTable = table(identifier);
-if (existingTable == null) {
+IcebergContent existingContent = fetchContent(identifier);
+
+if (existingContent == null || existingContent.getType() != type) {
Review Comment:
updated throw an exception that content type is not matching.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] feat: Implement create table and update table api for rest catalog. [iceberg-rust]
liurenjie1024 commented on code in PR #97:
URL: https://github.com/apache/iceberg-rust/pull/97#discussion_r1405543566
##
crates/catalog/rest/src/catalog.rs:
##
@@ -1017,31 +1109,31 @@ mod tests {
.with_summary(Summary {
operation: Operation::Append,
other: HashMap::from_iter([
- ("spark.app.id",
"local-1646787004168"),
- ("added-data-files", "1"),
- ("added-records", "1"),
- ("added-files-size", "697"),
- ("changed-partition-count", "1"),
- ("total-records", "1"),
- ("total-files-size", "697"),
- ("total-data-files", "1"),
- ("total-delete-files", "0"),
- ("total-position-deletes", "0"),
- ("total-equality-deletes", "0")
-].iter().map(|p|(p.0.to_string(), p.1.to_string(
+("spark.app.id", "local-1646787004168"),
Review Comment:
Seems rust-fmt not enforcing the indentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] feat: Implement create table and update table api for rest catalog. [iceberg-rust]
liurenjie1024 commented on code in PR #97:
URL: https://github.com/apache/iceberg-rust/pull/97#discussion_r1405543566
##
crates/catalog/rest/src/catalog.rs:
##
@@ -1017,31 +1109,31 @@ mod tests {
.with_summary(Summary {
operation: Operation::Append,
other: HashMap::from_iter([
- ("spark.app.id",
"local-1646787004168"),
- ("added-data-files", "1"),
- ("added-records", "1"),
- ("added-files-size", "697"),
- ("changed-partition-count", "1"),
- ("total-records", "1"),
- ("total-files-size", "697"),
- ("total-data-files", "1"),
- ("total-delete-files", "0"),
- ("total-position-deletes", "0"),
- ("total-equality-deletes", "0")
-].iter().map(|p|(p.0.to_string(), p.1.to_string(
+("spark.app.id", "local-1646787004168"),
Review Comment:
Seems rust-fmt not enforcing the indentation, let me do some check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] Spec: Clarify partition equality [iceberg]
emkornfield commented on code in PR #9125: URL: https://github.com/apache/iceberg/pull/9125#discussion_r1405545902 ## format/spec.md: ## @@ -607,6 +614,8 @@ Notes: 1. An alternative, *strict projection*, creates a partition predicate that will match a file if all of the rows in the file must match the scan predicate. These projections are used to calculate the residual predicates for each file in a scan. 2. For example, if `file_a` has rows with `id` between 1 and 10 and a delete file contains rows with `id` between 1 and 4, a scan for `id = 9` may ignore the delete file because none of the deletes can match a row that will be selected. +3. Floating point partition values are considered equal if their IEEE 754 floating-point “single format” bit layout +are equal with NaNs normalized to have only the the most significant mantissa bit set (the equivelant of calling `Float.floatToIntBits` or `Double.doubleToLongBits` in Java). The Avro specification requires all all floating point values are encoded in this format. Review Comment: If we don't define NaN values as equal, as I read the spec there is no way to specify positition delete files for values that have an NaN partition value -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] Spec: Clarify partition equality [iceberg]
emkornfield commented on code in PR #9125: URL: https://github.com/apache/iceberg/pull/9125#discussion_r1405545902 ## format/spec.md: ## @@ -607,6 +614,8 @@ Notes: 1. An alternative, *strict projection*, creates a partition predicate that will match a file if all of the rows in the file must match the scan predicate. These projections are used to calculate the residual predicates for each file in a scan. 2. For example, if `file_a` has rows with `id` between 1 and 10 and a delete file contains rows with `id` between 1 and 4, a scan for `id = 9` may ignore the delete file because none of the deletes can match a row that will be selected. +3. Floating point partition values are considered equal if their IEEE 754 floating-point “single format” bit layout +are equal with NaNs normalized to have only the the most significant mantissa bit set (the equivelant of calling `Float.floatToIntBits` or `Double.doubleToLongBits` in Java). The Avro specification requires all all floating point values are encoded in this format. Review Comment: If we don't define NaN values as equal, as I read the spec there is no way to specify positition delete files for values that have an NaN partition value, if that is already the case can be I can change this to match the reference implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] Spec: Clarify partition equality [iceberg]
emkornfield commented on code in PR #9125: URL: https://github.com/apache/iceberg/pull/9125#discussion_r1405547938 ## format/spec.md: ## @@ -305,6 +305,13 @@ The source column, selected by id, must be a primitive type and cannot be contai Partition specs capture the transform from table data to partition values. This is used to transform predicates to partition predicates, in addition to transforming data values. Deriving partition predicates from column predicates on the table data is used to separate the logical queries from physical storage: the partitioning can change and the correct partition filters are always derived from column predicates. This simplifies queries because users don’t have to supply both logical predicates and partition predicates. For more information, see Scan Planning below. +Two partition specs are considered equivalent with each other if they have the same number of fields +and for each corresponding field, the fields have the same source column ID, transform definition +and partition name. Writers must not create a new parition spec if there already exists a compatible partition +spec defined in the table. Review Comment: I believe this is done now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] Spec: Clarify partition equality [iceberg]
emkornfield commented on code in PR #9125: URL: https://github.com/apache/iceberg/pull/9125#discussion_r1405548509 ## format/spec.md: ## @@ -305,6 +305,10 @@ The source column, selected by id, must be a primitive type and cannot be contai Partition specs capture the transform from table data to partition values. This is used to transform predicates to partition predicates, in addition to transforming data values. Deriving partition predicates from column predicates on the table data is used to separate the logical queries from physical storage: the partitioning can change and the correct partition filters are always derived from column predicates. This simplifies queries because users don’t have to supply both logical predicates and partition predicates. For more information, see Scan Planning below. +Two partition specs are considered compatible with each other if they have the same number of fields Review Comment: updated. ## format/spec.md: ## @@ -607,6 +614,8 @@ Notes: 1. An alternative, *strict projection*, creates a partition predicate that will match a file if all of the rows in the file must match the scan predicate. These projections are used to calculate the residual predicates for each file in a scan. 2. For example, if `file_a` has rows with `id` between 1 and 10 and a delete file contains rows with `id` between 1 and 4, a scan for `id = 9` may ignore the delete file because none of the deletes can match a row that will be selected. +3. Floating point partition values are considered equal if their IEEE 754 floating-point “single format” bit layout Review Comment: should be fixed now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] Spec: Clarify partition equality [iceberg]
emkornfield commented on code in PR #9125: URL: https://github.com/apache/iceberg/pull/9125#discussion_r1405548802 ## format/spec.md: ## @@ -305,6 +305,13 @@ The source column, selected by id, must be a primitive type and cannot be contai Partition specs capture the transform from table data to partition values. This is used to transform predicates to partition predicates, in addition to transforming data values. Deriving partition predicates from column predicates on the table data is used to separate the logical queries from physical storage: the partitioning can change and the correct partition filters are always derived from column predicates. This simplifies queries because users don’t have to supply both logical predicates and partition predicates. For more information, see Scan Planning below. +Two partition specs are considered equivalent with each other if they have the same number of fields +and for each corresponding field, the fields have the same source column ID, transform definition +and partition name. Writers must not create a new parition spec if there already exists a compatible partition +spec defined in the table. Review Comment: I believe this should be fixed now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] Question about "Column ID" in Parquet Sepc [iceberg]
emkornfield commented on issue #9099: URL: https://github.com/apache/iceberg/issues/9099#issuecomment-1827021236 I can make a PR to clarify -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] refactor: Removing type infromation in `Struct` literal. [iceberg-rust]
liurenjie1024 commented on code in PR #103:
URL: https://github.com/apache/iceberg-rust/pull/103#discussion_r1405571325
##
crates/iceberg/src/spec/values.rs:
##
@@ -554,13 +554,14 @@ impl From<&Literal> for JsonValue {
PrimitiveLiteral::Decimal(_) => todo!(),
},
Literal::Struct(s) => {
Review Comment:
Yes, after removing this, directly ser/de from json no longer works. The
correct way to do this is to ser/de is using the ser/de module, you can see
[this
method](https://github.com/apache/iceberg-rust/blob/28d7006ba3af12b1f99dc078780b08ccdd8e93b1/crates/iceberg/src/spec/values.rs#L1654)
as example. It delegates ser/de to serializaiton/deserialization system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] chore(deps): Update uuid requirement from ~1.5.0 to ~1.6.1 [iceberg-rust]
dependabot[bot] commented on PR #107: URL: https://github.com/apache/iceberg-rust/pull/107#issuecomment-1827057231 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting `@dependabot ignore this major version` or `@dependabot ignore this minor version`. You can also ignore all major, minor, or patch releases for a dependency by adding an [`ignore` condition](https://docs.github.com/en/code-security/supply-chain-security/configuration-options-for-dependency-updates#ignore) with the desired `update_types` to your config file. If you change your mind, just re-open this PR and I'll resolve any conflicts on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] chore(deps): Update uuid requirement from ~1.5.0 to ~1.6.1 [iceberg-rust]
liurenjie1024 commented on PR #107: URL: https://github.com/apache/iceberg-rust/pull/107#issuecomment-1827057209 This dependabot didn't remove `~` prefix, I'll close this pr and upgrade it by hand. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] chore(deps): Update uuid requirement from ~1.5.0 to ~1.6.1 [iceberg-rust]
liurenjie1024 closed pull request #107: chore(deps): Update uuid requirement from ~1.5.0 to ~1.6.1 URL: https://github.com/apache/iceberg-rust/pull/107 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] feat: Add support for `UnboundPartitionSpec`. [iceberg-rust]
liurenjie1024 commented on issue #98: URL: https://github.com/apache/iceberg-rust/issues/98#issuecomment-1827058935 > I think I'm still misunderstanding the UnboundPartitionSpec binding process, especially determining the spec_id during committing transaction. I opened a draft pr，please take a look and leave your comments when you are free! thanks， @liurenjie1024 Cool, I'll take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] feat: support UnboundPartitionSpec [iceberg-rust]
liurenjie1024 commented on code in PR #106:
URL: https://github.com/apache/iceberg-rust/pull/106#discussion_r1405573783
##
crates/iceberg/src/error.rs:
##
@@ -33,6 +33,8 @@ pub enum ErrorKind {
/// service error.
Unexpected,
+/// Iceberg finds some conflict when checking.
+Conflict,
Review Comment:
I think this is same as `InvalidData`?
##
crates/iceberg/src/spec/partition.rs:
##
@@ -60,6 +62,99 @@ impl PartitionSpec {
}
}
+static PARTITION_DATA_ID_START: i32 = 1000;
+
+/// Reference to [`UnboundPartitionSpec`].
+pub type UnboundPartitionSpecRef = Arc;
+#[derive(Debug, Serialize, Deserialize, PartialEq, Eq, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Unbound partition field can be built without a schema and later bound to a
schema.
+pub struct UnboundPartitionField {
+/// A source column id from the table’s schema
+pub source_id: i32,
+/// A partition field id that is used to identify a partition field and is
unique within a partition spec.
+/// In v2 table metadata, it is unique across all partition specs.
+pub partition_id: Option,
+/// A partition name.
+pub name: String,
+/// A transform that is applied to the source column to produce a
partition value.
+pub transform: Transform,
+}
+
+#[derive(Debug, Serialize, Deserialize, PartialEq, Eq, Clone, Default,
Builder)]
Review Comment:
Use `TypedBuilder`?
##
crates/iceberg/src/spec/partition.rs:
##
@@ -60,6 +62,99 @@ impl PartitionSpec {
}
}
+static PARTITION_DATA_ID_START: i32 = 1000;
+
+/// Reference to [`UnboundPartitionSpec`].
+pub type UnboundPartitionSpecRef = Arc;
+#[derive(Debug, Serialize, Deserialize, PartialEq, Eq, Clone)]
+#[serde(rename_all = "kebab-case")]
+/// Unbound partition field can be built without a schema and later bound to a
schema.
+pub struct UnboundPartitionField {
+/// A source column id from the table’s schema
+pub source_id: i32,
+/// A partition field id that is used to identify a partition field and is
unique within a partition spec.
+/// In v2 table metadata, it is unique across all partition specs.
+pub partition_id: Option,
+/// A partition name.
+pub name: String,
+/// A transform that is applied to the source column to produce a
partition value.
+pub transform: Transform,
+}
+
+#[derive(Debug, Serialize, Deserialize, PartialEq, Eq, Clone, Default,
Builder)]
+#[serde(rename_all = "kebab-case")]
+#[builder(setter(prefix = "with"))]
+/// Unbound partition spec can be built without a schema and later bound to a
schema.
+pub struct UnboundPartitionSpec {
+/// Identifier for PartitionSpec
+pub spec_id: Option,
+/// Details of the partition spec
+#[builder(setter(each(name = "with_unbound_partition_field")))]
+pub fields: Vec,
+}
+
+impl UnboundPartitionSpec {
+/// last assigned id for partitioned field
+pub fn unpartitioned_last_assigned_id() -> i32 {
+PARTITION_DATA_ID_START - 1
+}
+
+/// Create unbound partition spec builer
+pub fn builder() -> UnboundPartitionSpecBuilder {
+UnboundPartitionSpecBuilder::default()
+}
+
+/// Bind unbound partition spec to a schema
+pub fn bind(&self, schema: SchemaRef) -> Result {
Review Comment:
In fact, I'm hesitating to implement `bind` method for now since I don't
know if it's really useful, maybe we should implement it later when necessary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [I] Expose PyIceberg table as PyArrow Dataset [iceberg-python]
RahulDubey391 commented on issue #30: URL: https://github.com/apache/iceberg-python/issues/30#issuecomment-1827142768 Hi @Fokko , I can have a look into the issue! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
Re: [PR] Flink missing backports [iceberg]
pvary commented on PR #9144: URL: https://github.com/apache/iceberg/pull/9144#issuecomment-1827315584 Thanks @stevenzwu for the merge! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
