sungwy closed issue #368: Support setting a snapshot property in same commit as
spark.sql
URL: https://github.com/apache/iceberg-python/issues/368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
sungwy commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-2368148002
Snapshot property can now be specified in PyIceberg Table APIs
https://github.com/apache/iceberg-python/pull/419
--
This is an automated message from the Apache Git Servi
github-actions[bot] commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-2367035423
This issue has been automatically marked as stale because it has been open
for 180 days with no activity. It will be closed in next 14 days if no further
activity oc
syun64 commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-2021532558
> In order to implement this with snapshot properties I want my writer to do
the following transactionally:
> Fetch the current snapshot's dateranges property.
Modify th
brianfromoregon commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-2020990035
Hi @syun64, thanks for chiming in!
My batch app store historical data, there is always a date column. It runs
for each date and will insert data for that date. So
syun64 commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-2020762590
Hi @brianfromoregon and @corleyma , from my understanding of PyIceberg and
PySpark Iceberg, I'm not sure if allowing the two separate clients to
participate in the same transacti
brianfromoregon commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1988935623
Hi @corleyma, my thinking was that Issue 367 is meant to represent
"Interpretation 1" and this issue 368 is meant to represent "Interpretation 2".
Fully agreed that both
Gowthami03B commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1944327908
https://github.com/apache/iceberg-python/pull/419
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use th
Gowthami03B commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1940343932
@brianfromoregon @Fokko can I take a stab at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
Fokko commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1933869341
I would love that, and this is what I suggested in
https://github.com/apache/iceberg-python/issues/368#issuecomment-1928020308
--
This is an automated message from the Apache Gi
ajosephides commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1933712252
As @brianfromoregon has mentioned I also understood the issue raised to "
represent extending the API to allow same commit semantics like the java"
--
This is an automated
brianfromoregon commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1930565915
Ok agreed. So my intention was to have this issue represent extending the
API to allow same commit semantics like the java cookbook, and then issue #367
represent the (s
Fokko commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1930561761
Because it is not part of the API, so we need to extend it :) In Python, you
would append an Arrow table to the Iceberg table and set the properties in the
same commit (snapshot).
brianfromoregon commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1930559264
@Fokko Yes I am using python. So this is possible from java but impossible
from python, interesting I wonder why.
--
This is an automated message from the Apache Git S
Fokko commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1930557451
@brianfromoregon In the cookbook example it will be in the same commit,
which will result into a single snapshot. I was under the impression that you
also want to replicate this o
brianfromoregon commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1930549081
@Fokko Interesting, that makes sense, so what does the linked cookbook code
mean when it says "in the same commit"?
--
This is an automated message from the Apache Git
Fokko commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1930546610
@brianfromoregon That's not possible. Spark will create the snapshot, and
those are immutable. So you cannot update those afterward in PyIceberg.
--
This is an automated message
brianfromoregon commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1930497796
Beyond writing snapshot summary fields, this issue is also requesting
ability to write those fields in same snapshot as one created by spark.sql.
That would take changes
Fokko commented on issue #368:
URL: https://github.com/apache/iceberg-python/issues/368#issuecomment-1928020308
Thanks for raising this @brianfromoregon!
I think it would be a great addition. We need to extend the `.append` and
`.overwrite` API and allow passing in a map. And then it
brianfromoregon opened a new issue, #368:
URL: https://github.com/apache/iceberg-python/issues/368
### Feature Request / Improvement
This
[cookbook](https://tabular.io/apache-iceberg-cookbook/data-engineering-incremental-processing/)
has a java snippet to update a snapshot property a
20 matches
Mail list logo
