Re: [I] Data loss bug in MergeIntoCommand [iceberg]

2025-01-01 Thread via GitHub
BsoBird closed issue #11765: Data loss bug in MergeIntoCommand URL: https://github.com/apache/iceberg/issues/11765 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [I] Data loss bug in MergeIntoCommand [iceberg]

2024-12-20 Thread via GitHub
rdblue commented on issue #11765: URL: https://github.com/apache/iceberg/issues/11765#issuecomment-2557395549 I just left a -1 on the docs PR. I don't think that this is the right place to put a warning and I also think that the warning is overly broad and would lead to confusion. -- Thi

Re: [I] Data loss bug in MergeIntoCommand [iceberg]

2024-12-13 Thread via GitHub
BsoBird commented on issue #11765: URL: https://github.com/apache/iceberg/issues/11765#issuecomment-2542688203 @RussellSpitzer Sir.If you have time, please review this PR for me. I believe we need to warn users against doing this in the documentation. https://github.com/apache/iceberg

Re: [I] Data loss bug in MergeIntoCommand [iceberg]

2024-12-13 Thread via GitHub
RussellSpitzer commented on issue #11765: URL: https://github.com/apache/iceberg/issues/11765#issuecomment-2542309392 > However, sir, I might have discovered some issues. When executing the COW-MERGE-INTO command, Spark needs to use the ods_table twice. The first time is to match data

Re: [I] Data loss bug in MergeIntoCommand [iceberg]

2024-12-13 Thread via GitHub
BsoBird commented on issue #11765: URL: https://github.com/apache/iceberg/issues/11765#issuecomment-2541708695 @RussellSpitzer Sir. I am using Spark version 3.5.1, and the Iceberg version is 1.7.1/1.6.1. Sql: ``` merge into target_iceberg_table t using ( select

Re: [I] Data loss bug in MergeIntoCommand [iceberg]

2024-12-13 Thread via GitHub
RussellSpitzer commented on issue #11765: URL: https://github.com/apache/iceberg/issues/11765#issuecomment-2541339227 It would also be very helpful to know how you are determining there is data loss -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [I] Data loss bug in MergeIntoCommand [iceberg]

2024-12-13 Thread via GitHub
RussellSpitzer commented on issue #11765: URL: https://github.com/apache/iceberg/issues/11765#issuecomment-2541337509 You'll need to elaborate a bit more. What does "losing some of the increment new data writes" mean? Can you give an example? We can't really debug the generic case si

Re: [I] Data loss bug in MergeIntoCommand [iceberg]

2024-12-12 Thread via GitHub
BsoBird commented on issue #11765: URL: https://github.com/apache/iceberg/issues/11765#issuecomment-2540547547 > Shouldn't be any problem i'm aware of. If there are incorrectly applied merges in that situation it would be the validations at commit time are failing. Sir, currently it a

Re: [I] Data loss bug in MergeIntoCommand [iceberg]

2024-12-12 Thread via GitHub
sfc-gh-rspitzer commented on issue #11765: URL: https://github.com/apache/iceberg/issues/11765#issuecomment-2540419261 Shouldn't be any problem i'm aware of. If there are incorrectly applied merges in that situation it would be the validations at commit time are failing. -- This is an a

Re: [I] Data loss bug in MergeIntoCommand [iceberg]

2024-12-12 Thread via GitHub
BsoBird commented on issue #11765: URL: https://github.com/apache/iceberg/issues/11765#issuecomment-2540372837 > We don't have auto merge schema so I don't think we have the same issue as in the Delta issue (at least not yet). Do you have any more details about the data loss? Sir, if

Re: [I] Data loss bug in MergeIntoCommand [iceberg]

2024-12-12 Thread via GitHub
RussellSpitzer commented on issue #11765: URL: https://github.com/apache/iceberg/issues/11765#issuecomment-2539320431 We don't have auto merge schema so I don't think we have the same issue as in the Delta issue (at least not yet). Do you have any more details about the data loss? -- Thi