amogh-jahagirdar commented on issue #6783: URL: https://github.com/apache/iceberg/issues/6783#issuecomment-1424617561
> I have another job that works just fine and I took a look at the inputFormat, outputFormat, serialization.lib for the iceberg table that is working and they aren't set either. That's my point, sorry for the confusion. These properties should **not** have to be set. >That job is saving to a different database and table but it's also reading from a non iceberg table within the catalog using glueContext.create_dynamic_frame.from_catalog(). It then writes using the same SQL statements as above but to a different location and it works fine. I do notice that within the table properties of the table that is working it has metadata_location, previous_metadata_location, and table_type where as the job that fails only has metadata_location, table_type. The previous metadata location just means there was at least 2 commits on the table including the create. Which makes sense given your script. My point is more that it's expected that every commit of the Glue catalog does not retain these Hive serde/input/output format properties. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
