This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d1ef991bbd93 [SPARK-49798][DOC] Fix inaccurate documentation of
RuntimeConfig.get
d1ef991bbd93 is described below
commit d1ef991bbd935a38cc5550d5a3fc0a54c260eeb8
Author: BRIJ RAJ KISHORE <[email protected]>
AuthorDate: Thu Jun 4 22:29:27 2026 +0800
[SPARK-49798][DOC] Fix inaccurate documentation of RuntimeConfig.get
### What changes were proposed in this pull request?
Clarify the Scaladoc and PySpark docstring for three public `RuntimeConfig`
methods whose descriptions did not accurately explain the interaction between
an explicitly-set value and the key's built-in (`ConfigEntry`) default value.
**`get(key: String)` / `get(key)`**
- Before: *"If the key is not set yet, return its default value if
possible"*
- After: *"If the key is not explicitly set, return its built-in default
value if one exists"*
**`get(key: String, default: String)` / `get(key, default)`**
- Before: *"If the key is not set yet, return the user given `default`"* —
silent about overriding the ConfigEntry default
- After: explicitly states the user-supplied `default` is returned *instead
of* the key's built-in default value
**`getOption(key: String)`**
- Before: *"return its default value if possible, otherwise `None`"* — vague
- After: *"return `Some` of its built-in default value if one exists,
otherwise `None`"*
**PySpark `RuntimeConfig.get()`**
- Removed the misleading phrase *"assuming it is set"*
- Rewrote the description to explain both calling forms separately
- Added a `Raises` section documenting `SparkNoSuchElementException`
- Fixed the `Returns` section to follow NumPy docstring format
- Added four illustrative doctest examples using
`spark.sql.sources.partitionOverwriteMode` that demonstrate the
built-in-default behaviour
---
### Why are the changes needed?
The documentation was misleading in a subtle but important way. Users
reading `get(key)` would think it throws whenever the key is *unset*, but in
fact it returns the ConfigEntry's built-in default silently. Similarly, users
reading `get(key, default)` had no way to know the supplied default overrides
the built-in default — not just fills a gap when no default exists at all.
A concrete example (from SPARK-49798):
```python
spark.conf.unset("spark.sql.session.timeZone")
# Old doc implied this would throw — it does NOT; returns built-in default
"Etc/UTC"
spark.conf.get("spark.sql.session.timeZone")
# => "Etc/UTC"
# Old doc did not mention this ignores the built-in default "Etc/UTC"
spark.conf.get("spark.sql.session.timeZone", "Europe/Berlin")
# => "Europe/Berlin"
```
---
### Does this PR introduce _any_ user-facing change?
No.
---
### How was this patch tested?
No new tests added. The described behaviour is already fully covered by
existing tests:
- **Scala:** `RuntimeConfigSuite` — *"set and get a config with
defaultValue"* tests all three cases (`get(key)` returning built-in default,
`getOption(key)` returning `Some(builtInDefault)`, and `get(key, userDefault)`
overriding the built-in default)
- **Python:** `pyspark/sql/tests/test_conf.py` — `test_conf` (lines 38–45)
asserts the same behaviour using `spark.sql.sources.partitionOverwriteMode`
Four new doctest examples were added to `pyspark/sql/conf.py`. They are
executable via `python python/pyspark/sql/conf.py` and assert the same cases
the unit tests cover.
---
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.6
Closes #56154 from brijrajk/SPARK-49798-fix-runtimeconfig-docs.
Authored-by: BRIJ RAJ KISHORE <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
python/pyspark/sql/conf.py | 36 +++++++++++++++++++---
.../scala/org/apache/spark/sql/RuntimeConfig.scala | 16 +++++-----
2 files changed, 41 insertions(+), 11 deletions(-)
diff --git a/python/pyspark/sql/conf.py b/python/pyspark/sql/conf.py
index 8e4fa1073fb6..d4991cd2a412 100644
--- a/python/pyspark/sql/conf.py
+++ b/python/pyspark/sql/conf.py
@@ -62,8 +62,12 @@ class RuntimeConfig:
self, key: str, default: Union[Optional[str], _NoValueType] = _NoValue
) -> Optional[str]:
"""
- Returns the value of Spark runtime configuration property for the
given key,
- assuming it is set.
+ Returns the value of Spark runtime configuration property for the
given key.
+
+ If ``default`` is not provided and the key is not explicitly set,
returns the key's
+ built-in default value if one exists, otherwise raises an exception.
If ``default``
+ is provided and the key is not explicitly set, returns ``default``
instead of
+ the key's built-in default value (if any).
.. versionadded:: 2.0.0
@@ -72,19 +76,43 @@ class RuntimeConfig:
key : str
key of the configuration to get.
default : str, optional
- value of the configuration to get if the key does not exist.
+ value to return if the key is not explicitly set. When provided,
this overrides
+ the key's built-in default value.
Returns
-------
- The string value of the configuration set, or None.
+ str or None
+ The value of the configuration property.
+
+ Raises
+ ------
+ pyspark.errors.SparkNoSuchElementException
+ If the key is not explicitly set, has no built-in default value,
and ``default``
+ is not provided.
Examples
--------
+ A key with no built-in default returns the provided ``default`` when
not explicitly set:
+
>>> spark.conf.get("non-existent-key", "my_default")
'my_default'
+
+ An explicitly set key returns its value:
+
>>> spark.conf.set("my_key", "my_value")
>>> spark.conf.get("my_key")
'my_value'
+
+ A key with a built-in default returns that default when not explicitly
set:
+
+ >>> spark.conf.unset("spark.sql.sources.partitionOverwriteMode")
+ >>> spark.conf.get("spark.sql.sources.partitionOverwriteMode")
+ 'STATIC'
+
+ Providing ``default`` overrides the built-in default, not just the
absence of a value:
+
+ >>> spark.conf.get("spark.sql.sources.partitionOverwriteMode",
"DYNAMIC")
+ 'DYNAMIC'
"""
self._check_type(key, "key")
if default is _NoValue:
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala
b/sql/api/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala
index 7ea5fe9bfb10..4cf7c324a406 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala
@@ -61,20 +61,21 @@ abstract class RuntimeConfig {
/**
* Returns the value of Spark runtime configuration property for the given
key. If the key is
- * not set yet, return its default value if possible, otherwise
`NoSuchElementException` will be
- * thrown.
+ * not explicitly set, return its built-in default value if one exists,
otherwise
+ * `NoSuchElementException` will be thrown.
*
* @throws java.util.NoSuchElementException
- * if the key is not set and does not have a default value
+ * if the key is not set and does not have a built-in default value
* @since 2.0.0
*/
- @throws[NoSuchElementException]("if the key is not set and there is no
default value")
+ @throws[NoSuchElementException]("if the key is not set and there is no
built-in default value")
def get(key: String): String
/**
* Returns the value of Spark runtime configuration property for the given
key. If the key is
- * not set yet, return the user given `default`. This is useful when its
default value defined
- * by Apache Spark is not the desired one.
+ * not explicitly set, return the user given `default` instead of the key's
built-in default
+ * value (if any). This is useful when the built-in default value defined by
Apache Spark is not
+ * the desired one.
*
* @since 2.0.0
*/
@@ -113,7 +114,8 @@ abstract class RuntimeConfig {
/**
* Returns the value of Spark runtime configuration property for the given
key. If the key is
- * not set yet, return its default value if possible, otherwise `None` will
be returned.
+ * not explicitly set, return `Some` of its built-in default value if one
exists, otherwise
+ * `None` will be returned.
*
* @since 2.0.0
*/
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
