Re: [I] Equality delete lost after compact data files [iceberg]

2024-11-13 Thread via GitHub
CodingJun commented on issue #10312: URL: https://github.com/apache/iceberg/issues/10312#issuecomment-2472793323 > yea I think @RussellSpitzer is right, we should rely on validation error to prevent this scenario here, ie T1 should not be able to commit successfully. > > i need to und

Re: [I] Equality delete lost after compact data files [iceberg]

2024-07-31 Thread via GitHub
szehon-ho commented on issue #10312: URL: https://github.com/apache/iceberg/issues/10312#issuecomment-2261085249 yea I think @RussellSpitzer is right, we should rely on validation error to prevent this scenario here, ie T1 should not be able to commit successfully. i need to understa

Re: [I] Equality delete lost after compact data files [iceberg]

2024-07-30 Thread via GitHub
pvary commented on issue #10312: URL: https://github.com/apache/iceberg/issues/10312#issuecomment-2259232342 @RussellSpitzer: The compaction started at t1 and it will replace files created before t1. The equality delete is created later at t2. So if we use the same sequence number for the n

Re: [I] Equality delete lost after compact data files [iceberg]

2024-07-29 Thread via GitHub
RussellSpitzer commented on issue #10312: URL: https://github.com/apache/iceberg/issues/10312#issuecomment-2256202203 The validation error looks correct. Since we shouldn't be able to do the rewrite if the equality delete has been written in the middle. When we se using starting sequ

Re: [I] Equality delete lost after compact data files [iceberg]

2024-07-29 Thread via GitHub
pvary commented on issue #10312: URL: https://github.com/apache/iceberg/issues/10312#issuecomment-2255792704 @szehon-ho: This is a potential bug with compaction and equality delete files. Could you please help me understand if this is a valid scenarion? -- This is an automated message fro

Re: [I] Equality delete lost after compact data files [iceberg]

2024-05-16 Thread via GitHub
CodingJun commented on issue #10312: URL: https://github.com/apache/iceberg/issues/10312#issuecomment-2114253464 Do you know if this is a bug? @RussellSpitzer @aokolnychyi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [I] Equality delete lost after compact data files [iceberg]

2024-05-11 Thread via GitHub
CodingJun commented on issue #10312: URL: https://github.com/apache/iceberg/issues/10312#issuecomment-2105616995 > your process is in `use-starting-sequence-number = true` ? I test with `use-starting-sequence-number = true` and compact failed(apache iceberg1.4.3): `Exception in thread "main

Re: [I] Equality delete lost after compact data files [iceberg]

2024-05-11 Thread via GitHub
lurnagao-dahua commented on issue #10312: URL: https://github.com/apache/iceberg/issues/10312#issuecomment-2105608645 your process is in `use-starting-sequence-number = true` ? I test with `use-starting-sequence-number = true` and compact failed: `Exception in thread "main" org.apache.

Re: [I] Equality delete lost after compact data files [iceberg]

2024-05-10 Thread via GitHub
pvary commented on issue #10312: URL: https://github.com/apache/iceberg/issues/10312#issuecomment-2105594944 @CodingJun: Your analysis seems correct to me. We need to take the `minDataSequenceNumber` and `startingSequenceNumber`. @RussellSpitzer and @aokolnychyi might know more. --