HonahX commented on code in PR #701:
URL: https://github.com/apache/iceberg-python/pull/701#discussion_r1591768312
##########
pyiceberg/catalog/hive.py:
##########
@@ -356,11 +371,46 @@ def _create_lock_request(self, database_name: str,
table_name: str) -> LockReque
return lock_request
- def _commit_table(self, table_request: CommitTableRequest) ->
CommitTableResponse:
+ def _wait_for_lock(
+ self, database_name: str, table_name: str, lockid: int, open_client:
Client, properties: Properties
+ ) -> LockResponse:
+ acquire_lock_timeout = (
+ properties.get(TableProperties.HIVE_ACQUIRE_LOCK_TIMEOUT_MS,
TableProperties.HIVE_ACQUIRE_LOCK_TIMEOUT_MS_DEFAULT)
+ ) / 1000.0
+ lock_check_min_wait_time = (
+ properties.get(TableProperties.HIVE_LOCK_CHECK_MIN_WAIT_MS,
TableProperties.HIVE_LOCK_CHECK_MIN_WAIT_MS_DEFAULT)
+ ) / 1000.0
+ lock_check_max_wait_time = (
+ properties.get(TableProperties.HIVE_LOCK_CHECK_MAX_WAIT_MS,
TableProperties.HIVE_LOCK_CHECK_MAX_WAIT_MS_DEFAULT)
+ ) / 1000.0
Review Comment:
We could use
[`PropertyUtil.property_as_int`](https://github.com/apache/iceberg-python/blob/d872245f143b919b25fda90a7b7c7fb0729a402a/pyiceberg/table/__init__.py#L243-L258)
to get the values. This utl methods throws error message when failing to parse
the property.
BTW: You would properly need `#type: ignore` at the end of each
`PropertyUti.property_as_int` to pass the linter.
##########
pyiceberg/catalog/hive.py:
##########
@@ -111,6 +122,13 @@
HIVE2_COMPATIBLE = "hive.hive2-compatible"
HIVE2_COMPATIBLE_DEFAULT = False
+DEFAULT_LOCK_CHECK_MIN_WAIT_TIME = 2
+DEFAULT_LOCK_CHECK_MAX_WAIT_TIME = 30
+DEFAULT_LOCK_CHECK_RETRIES = 5
+DEFAULT_LOCK_CHECK_MULTIPLIER = 2
Review Comment:
In java, they are in the
[`HadoopConfiguration`](https://iceberg.apache.org/docs/nightly/configuration/#hadoop-configuration).
The hadoop configuration for a hive catalog is set at the catalog-level:
https://github.com/apache/iceberg/blob/817a5e1be1616af77329965ac3742c14ca3ae116/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java#L630
[Metastore parse these properties from the hadoop
configuration](https://github.com/apache/iceberg/blob/e10453a10d3f9237032ba7e649808347c699f6a2/hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java#L110-L125)
I call them "CatalogProperties" since pyiceberg does not have hadoop
configuration. Sorry if that creates any confusion.
The only hive-lock-related config in `TableProperties` is
[`engine.hive.lock-enabled`](https://github.com/apache/iceberg/blob/e7999a194dc0f32fba8fb515c9108764ed6ba6b5/core/src/main/java/org/apache/iceberg/TableProperties.java#L319)
which can enable/disable lock for a single table. But that is for
https://issues.apache.org/jira/browse/HIVE-26882, see the "warn" section in the
end of
https://iceberg.apache.org/docs/nightly/configuration/#hadoop-configuration. I
don't think we need that in pyiceberg now.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
