Script 'mail_helper' called by obssrc
Hello community,
here is the log from the commit of package python-url-normalize for
openSUSE:Factory checked in at 2026-10-26 10:02:45
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comparing /work/SRC/openSUSE:Factory/python-url-normalize (Old)
and /work/SRC/openSUSE:Factory/.python-url-normalize.new.2252680 (New)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-url-normalize"
Mon Oct 26 10:02:45 2026 rev:6 rq:1392535 version:3.0.1
Changes:
--------
---
/work/SRC/openSUSE:Factory/python-url-normalize/python-url-normalize.changes
2026-04-26 21:14:28.345165712 +0200
+++
/work/SRC/openSUSE:Factory/.python-url-normalize.new.2252680/python-url-normalize.changes
2026-10-26 10:04:13.577342380 +0100
@@ -1,0 +2,12 @@
+Sun Oct 25 20:09:26 UTC 2026 - Dirk Müller <[email protected]>
+
+- update to 3.0.1:
+ * Normalization fixes change canonical output for some URLs,
+ including encoded delimiters, query plus signs, explicit
+ empty query values, and trailing spaces. If normalized URLs
+ are used as cache keys or deduplication identifiers, review
+ stored keys and re-normalize the original URLs where
+ available; information discarded by earlier normalization
+ cannot be recovered from those normalized values alone.
+
+-------------------------------------------------------------------
Old:
----
url-normalize-3.0.0.tar.gz
New:
----
url-normalize-3.0.1.tar.gz
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other differences:
------------------
++++++ python-url-normalize.spec ++++++
--- /var/tmp/diff_new_pack.pWA3b7/_old 2026-10-26 10:04:14.892397507 +0100
+++ /var/tmp/diff_new_pack.pWA3b7/_new 2026-10-26 10:04:14.895397633 +0100
@@ -18,7 +18,7 @@
%{?sle15_python_module_pythons}
Name: python-url-normalize
-Version: 3.0.0
+Version: 3.0.1
Release: 0
Summary: URL normalization for Python
License: MIT
++++++ url-normalize-3.0.0.tar.gz -> url-normalize-3.0.1.tar.gz ++++++
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/CHANGELOG.md
new/url-normalize-3.0.1/CHANGELOG.md
--- old/url-normalize-3.0.0/CHANGELOG.md 2026-04-25 02:23:36.000000000
+0200
+++ new/url-normalize-3.0.1/CHANGELOG.md 2026-09-22 23:31:41.000000000
+0200
@@ -7,6 +7,36 @@
## [Unreleased]
+## [3.0.1] - 2026-09-22
+
+### Upgrade notes
+
+Normalization fixes change canonical output for some URLs, including encoded
delimiters, query plus signs, explicit empty query values, and trailing spaces.
If normalized URLs are used as cache keys or deduplication identifiers, review
stored keys and re-normalize the original URLs where available; information
discarded by earlier normalization cannot be recovered from those normalized
values alone.
+
+### Fixed
+
+- Preserve invalid UTF-8 percent-encoded octets instead of replacing them with
U+FFFD; valid UTF-8 still normalizes to NFC.
+- Preserve percent-encoded reserved characters in paths, keeping values such
as `a%2Fb` distinct from `a/b`.
+- Preserve encoded plus signs (`%2B`) in query parameters and explicit empty
values such as `flag=`.
+- Normalize Unicode and unsafe characters in URL credentials while preserving
encoded delimiters; correctly separate credentials containing `@` from the host.
+- Correctly parse bracketed IPv6 hosts and ports, lowercase IPv6 addresses,
and preserve zone identifier case.
+- Recognize long and custom URL schemes, protocol-relative URLs, and bare
domains or `localhost` with numeric ports.
+- Remove leading whitespace and parser-ignored control characters before
applying default domains and detecting URL schemes, preserving the intended
destination.
+- Trim trailing authority whitespace without discarding trailing spaces or
punctuation belonging to paths, query values, or fragments.
+- Convert hashbang fragments (`#!`) into a single escaped query parameter,
preserving existing parameters and leaving ordinary fragments intact.
+- Match query parameter allowlists against normalized hosts and equivalent
custom domain keys, including Unicode/Punycode aliases, case variants, and
trailing dots. Exact canonical keys take precedence over aliases.
+- Decode IDNA host output as ASCII regardless of the `charset` argument.
+- Keep encoded backslashes and C0/C1 control characters encoded in
`url_humanize()` output, and retain the normalized URL when a decoding
candidate cannot be parsed.
+- Resolve query parameter allowlists once per URL to avoid repeated domain
alias scans for each parameter.
+
+### Documentation
+
+- Correct normalization examples and clarify that parameter filtering uses
allowlists, unknown domains have no default allowed parameters, and Unicode URL
data uses UTF-8 percent encoding regardless of `charset`.
+
+### Internal
+
+- Enforce 100% statement coverage in the test suite and expand regression
coverage for URL component handling.
+
## [3.0.0] - 2026-04-24
### Added
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/README.md
new/url-normalize-3.0.1/README.md
--- old/url-normalize-3.0.0/README.md 2026-04-25 02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/README.md 2026-09-22 23:31:41.000000000 +0200
@@ -37,7 +37,7 @@
- **Versatile URL Handling**: Handles empty strings, double-slash URLs
(//domain.tld), and shebang (#!) URLs.
- **Developer Friendly**:
- Python 3.10+ compatibility.
- - 100% test coverage.
+ - 100% statement coverage, enforced by the test suite.
- Modern type hints and string handling.
Inspired by Sam Ruby's
[urlnorm.py](http://intertwingly.net/blog/2004/08/04/Urlnorm).
@@ -67,19 +67,23 @@
# Basic normalization (uses https by default)
print(url_normalize("www.foo.com:80/foo"))
-# Output: https://www.foo.com/foo
+# Output: https://www.foo.com:80/foo
# With custom default scheme
print(url_normalize("www.foo.com/foo", default_scheme="http"))
# Output: http://www.foo.com/foo
```
+The `charset` argument remains for compatibility. Unicode characters use UTF-8
percent encoding, regardless of this argument.
+
#### Query Parameter Filtering
-You can strip out tracking parameters and only keep the ones you care about
using allowlists.
+With `filter_params=True`, normalization retains only allowlisted query
parameters. The built-in rules cover selected domains.
+
+Other domains have an empty default allowlist. Without a custom allowlist,
filtering removes all query parameters from those domains.
```python
-# With query parameter filtering enabled (strips all params by default)
+# With the built-in Google allowlist
print(url_normalize("www.google.com/search?q=test&utm_source=test",
filter_params=True))
# Output: https://www.google.com/search?q=test
@@ -89,7 +93,7 @@
filter_params=True,
param_allowlist=["page", "id"]
))
-# Output: https://example.com?page=1&id=123
+# Output: https://example.com/?page=1&id=123
# With domain-specific parameter allowlists
print(url_normalize(
@@ -97,7 +101,7 @@
filter_params=True,
param_allowlist={"example.com": ["page", "id"]}
))
-# Output: https://example.com?page=1&id=123
+# Output: https://example.com/?page=1&id=123
```
#### Default Domain & Scheme
@@ -136,7 +140,7 @@
```bash
$ url-normalize "www.foo.com:80/foo"
-# Output: https://www.foo.com/foo
+# Output: https://www.foo.com:80/foo
# With custom default scheme
$ url-normalize -s http "www.foo.com/foo"
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/pyproject.toml
new/url-normalize-3.0.1/pyproject.toml
--- old/url-normalize-3.0.0/pyproject.toml 2026-04-25 02:23:36.000000000
+0200
+++ new/url-normalize-3.0.1/pyproject.toml 2026-09-22 23:31:41.000000000
+0200
@@ -1,6 +1,6 @@
[project]
name = "url-normalize"
-version = "3.0.0"
+version = "3.0.1"
description = "URL normalization for Python"
authors = [{ name = "Nikolay Panov", email = "[email protected]" }]
license = "MIT"
@@ -73,6 +73,7 @@
addopts = [
"--cov-report=term-missing:skip-covered",
"--cov=url_normalize",
+ "--cov-fail-under=100",
"--disable-socket",
"-v",
]
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/tests/test_cli.py
new/url-normalize-3.0.1/tests/test_cli.py
--- old/url-normalize-3.0.0/tests/test_cli.py 2026-04-25 02:23:36.000000000
+0200
+++ new/url-normalize-3.0.1/tests/test_cli.py 2026-09-22 23:31:41.000000000
+0200
@@ -26,6 +26,38 @@
)
+def test_cli_help_describes_allowlist_filtering() -> None:
+ """Describe filtering as an allowlist, not a tracking-parameter
blocklist."""
+ result = run_cli("--help")
+ assert result.returncode == 0
+ help_text = " ".join(result.stdout.split())
+ assert "Keep only allowlisted query parameters." in help_text
+ assert "Unknown domains have an empty default allowlist." in help_text
+
+
+def test_cli_help_describes_fixed_utf8_encoding() -> None:
+ """Describe charset as a compatibility option for Unicode URL input."""
+ result = run_cli("--help")
+ assert result.returncode == 0
+ help_text = " ".join(result.stdout.split())
+ assert "Retained for compatibility." in help_text
+ assert "Unicode characters use UTF-8 percent encoding." in help_text
+ normalized = run_cli("--charset", "iso-8859-1", "https://example.com/é";)
+ assert normalized.returncode == 0
+ assert normalized.stdout.strip() == "https://example.com/%C3%A9";
+
+
+def test_cli_successful_main(capsys, monkeypatch):
+ """Exercise successful CLI output in the measured process."""
+ monkeypatch.setattr(
+ sys, "argv", ["url-normalize", "http://EXAMPLE.com/./path/../other/";]
+ )
+ main()
+ captured = capsys.readouterr()
+ assert captured.out == "http://example.com/other/\n";
+ assert not captured.err
+
+
def test_cli_error_handling(capsys, monkeypatch):
"""Test CLI error handling when URL normalization fails."""
with patch("url_normalize.cli.url_normalize") as mock_normalize:
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/tests/test_deconstruct_url.py
new/url-normalize-3.0.1/tests/test_deconstruct_url.py
--- old/url-normalize-3.0.0/tests/test_deconstruct_url.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/tests/test_deconstruct_url.py 2026-09-22
23:31:41.000000000 +0200
@@ -2,6 +2,7 @@
import pytest
+from url_normalize import url_normalize
from url_normalize.tools import URL, deconstruct_url
@@ -38,3 +39,49 @@
"""Assert we got expected results from the deconstruct_url function."""
result = deconstruct_url(url)
assert result == expected, url
+
+
[email protected](
+ ("authority", "userinfo", "host", "port", "normalized_authority"),
+ [
+ ("[2001:DB8::1]:080", "", "[2001:DB8::1]", "080", "[2001:db8::1]"),
+ ("[2001:DB8::1]:0081", "", "[2001:DB8::1]", "0081",
"[2001:db8::1]:81"),
+ ("[::1]", "", "[::1]", "", "[::1]"),
+ ("[::1]:", "", "[::1]", "", "[::1]"),
+ ("user:pass@[::1]:080", "user:pass@", "[::1]", "080",
"user:pass@[::1]"),
+ ("EXAMPLE.com:080", "", "EXAMPLE.com", "080", "example.com"),
+ ("127.0.0.1:0081", "", "127.0.0.1", "0081", "127.0.0.1:81"),
+ ],
+)
+def test_deconstruct_url_separates_bracketed_host_and_port(
+ authority, userinfo, host, port, normalized_authority
+):
+ """Parse the entire bracketed host before considering a port separator."""
+ url = f"http://{authority}/path?q=1#fragment";
+ assert deconstruct_url(url) == URL(
+ "http", userinfo, host, port, "/path", "q=1", "fragment"
+ )
+ assert url_normalize(url) ==
f"http://{normalized_authority}/path?q=1#fragment";
+
+
[email protected]("whitespace", [" ", "\u00a0", " \t\r\n"])
[email protected]("suffix", ["", "/path ", "?q=x ", "#part "])
[email protected](
+ ("authority", "normalized"),
+ [
+ ("EXAMPLE.com", "example.com"),
+ ("example.com:443", "example.com"),
+ ("user:[email protected]:443", "user:[email protected]"),
+ ("[FE80::1%25ethA]:443", "[fe80::1%25ethA]"),
+ ],
+)
+def test_deconstruct_url_trims_authority_whitespace_only(
+ authority, normalized, whitespace, suffix
+):
+ """Trim the authority without deleting whitespace from other components."""
+ value = f"https://{authority}{whitespace}{suffix}";
+ assert deconstruct_url(value) ==
deconstruct_url(f"https://{authority}{suffix}";)
+ path_suffix = suffix if suffix.startswith("/") else "/" + suffix
+ assert (
+ url_normalize(value) == f"https://{normalized}{path_suffix.replace('
', '%20')}"
+ )
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/url-normalize-3.0.0/tests/test_generic_url_cleanup.py
new/url-normalize-3.0.1/tests/test_generic_url_cleanup.py
--- old/url-normalize-3.0.0/tests/test_generic_url_cleanup.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/tests/test_generic_url_cleanup.py 2026-09-22
23:31:41.000000000 +0200
@@ -2,9 +2,11 @@
from __future__ import annotations
+from urllib.parse import parse_qsl, urlsplit
+
import pytest
-from url_normalize.url_normalize import generic_url_cleanup
+from url_normalize.url_normalize import generic_url_cleanup, url_normalize
@pytest.mark.parametrize(
@@ -12,10 +14,59 @@
[
("//site/#!fragment", "//site/?_escaped_fragment_=fragment"),
("//site/page", "//site/page"),
- ("//site/?& ", "//site/"),
+ ("//site/?&&", "//site/"),
+ ("//site/?& ", "//site/? "),
],
)
def test_generic_url_cleanup_result_is_expected(url: str, expected: str) ->
None:
"""Assert we got expected results from the generic_url_cleanup function."""
result = generic_url_cleanup(url)
assert result == expected
+
+
[email protected](
+ ("suffix", "normalized_suffix"),
+ [
+ ("/a&", "/a&"),
+ ("/a ", "/a%20"),
+ ("/?q=why?", "/?q=why?"),
+ ("/?q=x ", "/?q=x%20"),
+ ("/#part&", "/#part%26"),
+ ("/#part?", "/#part%3F"),
+ ("/#part ", "/#part%20"),
+ ],
+)
+def test_generic_url_cleanup_preserves_component_data(suffix,
normalized_suffix):
+ """Preserve trailing punctuation and whitespace belonging to components."""
+ url = f"https://example.com{suffix}";
+ assert generic_url_cleanup(url) == url
+ assert url_normalize(url) == f"https://example.com{normalized_suffix}";
+
+
[email protected]("query", ["", "?q=x", "?q=x&", "?q=why?"])
[email protected](
+ ("fragment", "payload"),
+ [
+ ("!/route", "/route"),
+ ("!/route&a=b", "/route&a=b"),
+ ("!/C++%2B%26%23", "/C+++&#"),
+ ("!", ""),
+ ("!a#!b", "a#!b"),
+ ("part#!nested", None),
+ ("ordinary", None),
+ ],
+)
+def test_generic_url_cleanup_encodes_hashbang_as_one_parameter(
+ query, fragment, payload
+):
+ """Append the decoded hashbang payload as one value after existing
parameters."""
+ url = f"https://example.com/{query}#{fragment}";
+ if payload is None:
+ assert urlsplit(generic_url_cleanup(url)).fragment == fragment
+ return
+ expected = parse_qsl(query.removeprefix("?"), keep_blank_values=True)
+ expected.append(("_escaped_fragment_", payload))
+ for result in (generic_url_cleanup(url), url_normalize(url)):
+ parts = urlsplit(result)
+ assert parts.fragment == ""
+ assert parse_qsl(parts.query, keep_blank_values=True) == expected
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/tests/test_normalize_host.py
new/url-normalize-3.0.1/tests/test_normalize_host.py
--- old/url-normalize-3.0.0/tests/test_normalize_host.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/tests/test_normalize_host.py 2026-09-22
23:31:41.000000000 +0200
@@ -30,3 +30,29 @@
"""Assert we got expected results from the normalize_host function."""
result = normalize_host(host)
assert result == expected, host
+
+
[email protected]("charset", ["utf-8", "utf-16", "utf-32",
"iso-8859-1"])
[email protected](
+ ("host", "expected"),
+ [
+ ("пример.рф", "xn--e1afmkfd.xn--p1ai"),
+ ("EXAMPLE.COM", "example.com"),
+ ("under_score.example", "under_score.example"),
+ ("[2001:DB8::1]", "[2001:db8::1]"),
+ ],
+)
+def test_normalize_host_decodes_idna_output_as_ascii(host, expected, charset):
+ """Decode ASCII IDNA output independently of the input byte encoding."""
+ assert normalize_host(host, charset) == expected
+ assert normalize_host(expected.encode(charset), charset) == expected
+
+
[email protected]("zone", ["%25ethA", "%ethA", "%25Interface.10",
"%25ETH0"])
[email protected]("charset", ["utf-8", "utf-16"])
+def test_normalize_host_preserves_ipv6_zone_case(zone, charset):
+ """Lowercase the IPv6 address without changing its interface identifier."""
+ host = f"[FE80::ABCD{zone}]"
+ expected = f"[fe80::abcd{zone}]"
+ assert normalize_host(host, charset) == expected
+ assert normalize_host(host.encode(charset), charset) == expected
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/tests/test_normalize_path.py
new/url-normalize-3.0.1/tests/test_normalize_path.py
--- old/url-normalize-3.0.0/tests/test_normalize_path.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/tests/test_normalize_path.py 2026-09-22
23:31:41.000000000 +0200
@@ -2,7 +2,7 @@
import pytest
-from url_normalize.url_normalize import normalize_path
+from url_normalize.url_normalize import normalize_path, url_normalize
@pytest.mark.parametrize(
@@ -42,3 +42,17 @@
"""Assert we got expected results from the normalize_path function."""
result = normalize_path(path, "http")
assert result == expected, path
+
+
[email protected]("character", ":/?#[]@!$&'()*+,;=")
+def test_normalize_path_preserves_encoded_reserved_characters(character):
+ """Keep escaped delimiters distinct from literal path syntax."""
+ escaped = f"%{ord(character):02X}"
+ path = f"/a{escaped.lower()}b/%7e/e%CC%81/%25/%zz"
+ expected = f"/a{escaped}b/~/%C3%A9/%25/%25zz"
+ assert normalize_path(path, "https") == expected
+ url = f"https://example.com{expected}?x=1#end";
+ assert url_normalize(f"https://example.com{path}?x=1#end";) == url
+ assert url_normalize(url) == url
+ literal = "%23" if character == "#" else "%3F" if character == "?" else
character
+ assert normalize_path(f"/a{character}b", "https") == f"/a{literal}b"
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/tests/test_normalize_query.py
new/url-normalize-3.0.1/tests/test_normalize_query.py
--- old/url-normalize-3.0.0/tests/test_normalize_query.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/tests/test_normalize_query.py 2026-09-22
23:31:41.000000000 +0200
@@ -1,5 +1,7 @@
"""Tests for normalize_query function."""
+from urllib.parse import parse_qsl
+
import pytest
from url_normalize.url_normalize import normalize_query
@@ -22,3 +24,34 @@
"""Assert we got expected results from the normalize_query function."""
result = normalize_query(query)
assert result == expected, query
+
+
[email protected](
+ ("query", "expected"),
+ [
+ ("q=C%2b%2B", "q=C%2B%2B"),
+ ("C%2b%2B=value", "C%2B%2B=value"),
+ ("q=C++", "q=C++"),
+ ("q=C%20%2B+", "q=C%20%2B+"),
+ ("q=%252B", "q=%252B"),
+ ],
+)
+def test_normalize_query_preserves_form_plus_meaning(query, expected):
+ """Distinguish encoded plus signs, literal plus signs, and spaces."""
+ actual = normalize_query(query)
+ assert actual == expected
+ assert parse_qsl(actual) == parse_qsl(query)
+
+
[email protected]("query", ["flag=&flag", "=", "=value", "flag=",
"flag"])
[email protected]("filter_params", [False, True])
+def test_normalize_query_preserves_empty_value_separator(query, filter_params):
+ """Keep explicit empty values distinct from parameters without a value."""
+ actual = normalize_query(
+ query, filter_params=filter_params, param_allowlist=["flag", ""]
+ )
+ assert actual == query
+ if query == "flag=":
+ assert parse_qsl(actual, keep_blank_values=True, strict_parsing=True)
== [
+ ("flag", "")
+ ]
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/url-normalize-3.0.0/tests/test_normalize_query_filters.py
new/url-normalize-3.0.1/tests/test_normalize_query_filters.py
--- old/url-normalize-3.0.0/tests/test_normalize_query_filters.py
2026-04-25 02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/tests/test_normalize_query_filters.py
2026-09-22 23:31:41.000000000 +0200
@@ -2,9 +2,12 @@
from __future__ import annotations
+from unittest.mock import patch
+
import pytest
from url_normalize import url_normalize
+from url_normalize.param_allowlist import get_allowed_params
def test_param_filtering_disabled_by_default():
@@ -96,3 +99,128 @@
def test_parameter_filtering(url: str, expected: str):
"""Test URL parameter filtering functionality with various scenarios."""
assert url_normalize(url, filter_params=True) == expected
+
+
[email protected](
+ ("host", "canonical", "allowlist"),
+ [
+ ("www.google.com.", "www.google.com", None),
+ ("WWW.GOOGLE.COM.", "www.google.com", None),
+ ("www。google.com", "www.google.com", None),
+ ("google.com", "google.com", None),
+ ("пример.рф", "xn--e1afmkfd.xn--p1ai", {"xn--e1afmkfd.xn--p1ai":
["q"]}),
+ (
+ "xn--e1afmkfd.xn--p1ai.",
+ "xn--e1afmkfd.xn--p1ai",
+ {"xn--e1afmkfd.xn--p1ai": ["q"]},
+ ),
+ ("EXAMPLE.COM.", "example.com", ["q"]),
+ ],
+)
+def test_parameter_filtering_uses_normalized_host(host, canonical, allowlist):
+ """Use the output host's canonical spelling when choosing an allowlist."""
+ assert (
+ url_normalize(
+ f"https://{host}/?q=test&utm_source=x";,
+ filter_params=True,
+ param_allowlist=allowlist,
+ )
+ == f"https://{canonical}/?q=test";
+ )
+
+
[email protected]("port", ["", ":8080"])
+def test_parameter_filtering_preserves_bracketed_host(port):
+ """Keep IPv6 address colons intact when looking up a canonical host."""
+ assert (
+ url_normalize(
+ f"https://[2001:DB8::1]{port}/?q=test&utm_source=x";,
+ filter_params=True,
+ param_allowlist={"[2001:db8::1]": ["q"]},
+ )
+ == f"https://[2001:db8::1]{port}/?q=test";
+ )
+
+
[email protected]("host", ["пример.рф", "xn--e1afmkfd.xn--p1ai",
"ПРИМЕР.РФ."])
[email protected](
+ "key",
+ [
+ "пример.рф",
+ "xn--e1afmkfd.xn--p1ai",
+ "ПРИМЕР.РФ.",
+ "www。пример.рф",
+ "пример.рф:8080",
+ ],
+)
+def test_custom_allowlist_normalizes_mapping_keys(host, key):
+ """Match equivalent domain spellings on both sides of an allowlist."""
+ allowlist = {key: ["q"]}
+ result = url_normalize(
+ f"https://{host}/?q=1&utm_source=x";,
+ filter_params=True,
+ param_allowlist=allowlist,
+ )
+ assert result == "https://xn--e1afmkfd.xn--p1ai/?q=1";
+ assert allowlist == {key: ["q"]}
+
+
[email protected]("key", ["[2001:DB8::1]", "[2001:DB8::1]:8080"])
+def test_custom_allowlist_normalizes_bracketed_keys(key):
+ """Preserve IPv6 colons when normalizing dictionary keys."""
+ assert (
+ url_normalize(
+ "https://[2001:db8::1]/?q=1&drop=2";,
+ filter_params=True,
+ param_allowlist={key: ["q"]},
+ )
+ == "https://[2001:db8::1]/?q=1";
+ )
+
+
[email protected]("allowed", [[], ["q"]])
[email protected]("reverse", [False, True])
+def test_custom_allowlist_canonical_key_takes_precedence(allowed, reverse):
+ """Keep canonical rules authoritative instead of merging conflicting
aliases."""
+ entries = [("пример.рф", ["other"]), ("xn--e1afmkfd.xn--p1ai", allowed)]
+ allowlist = dict(reversed(entries) if reverse else entries)
+ expected = "https://xn--e1afmkfd.xn--p1ai/"; + ("?q=1" if allowed else "")
+ assert (
+ url_normalize(
+ "https://пример.рф/?q=1&other=2";,
+ filter_params=True,
+ param_allowlist=allowlist,
+ )
+ == expected
+ )
+
+
+def test_custom_allowlist_ignores_unrelated_invalid_domain():
+ """Keep a malformed unrelated key from breaking a valid domain lookup."""
+ assert (
+ url_normalize(
+ "https://пример.рф/?q=1";,
+ filter_params=True,
+ param_allowlist={"a" * 64 + ".example": [], "пример.рф": ["q"]},
+ )
+ == "https://xn--e1afmkfd.xn--p1ai/?q=1";
+ )
+
+
[email protected]("host", ["target.example", "unlisted.example"])
+def test_filtering_resolves_domain_rules_once_per_url(host):
+ """Bound domain lookups independently of how many parameters are
supplied."""
+ allowlist = {f"d{index}.example": ["q"] for index in range(20)}
+ allowlist["TARGET.EXAMPLE."] = ["q"]
+ query = "&".join(f"q={index}&drop={index}" for index in range(50))
+ with patch(
+ "url_normalize.normalize_query.get_allowed_params",
wraps=get_allowed_params
+ ) as lookup:
+ result = url_normalize(
+ f"https://{host}/?{query}";, filter_params=True,
param_allowlist=allowlist
+ )
+ expected = f"https://{host}/";
+ if host == "target.example":
+ expected += "?" + "&".join(f"q={index}" for index in range(50))
+ assert result == expected
+ lookup.assert_called_once_with(host, allowlist)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/tests/test_normalize_userinfo.py
new/url-normalize-3.0.1/tests/test_normalize_userinfo.py
--- old/url-normalize-3.0.0/tests/test_normalize_userinfo.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/tests/test_normalize_userinfo.py 2026-09-22
23:31:41.000000000 +0200
@@ -2,7 +2,7 @@
import pytest
-from url_normalize.url_normalize import normalize_userinfo
+from url_normalize.url_normalize import normalize_userinfo, url_normalize
@pytest.mark.parametrize(
@@ -19,3 +19,26 @@
"""Assert we got expected results from the normalize_userinfo function."""
result = normalize_userinfo(userinfo)
assert result == expected, userinfo
+
+
[email protected](
+ ("userinfo", "expected"),
+ [
+ ("us er:pa ss@", "us%20er:pa%20ss@"),
+ ("e\u0301:пароль@", "%C3%A9:%D0%BF%D0%B0%D1%80%D0%BE%D0%BB%D1%8C@"),
+ ("user:name:!$&'()*+,;=@", "user:name:!$&'()*+,;=@"),
+ ("user%3aname:p%40ss@", "user%3Aname:p%40ss@"),
+ ("us%65r:%ff@", "user:%FF@"),
+ ("user:50%@", "user:50%25@"),
+ ("user:p@ss@", "user:p%40ss@"),
+ ('user:p"<>\\@', "user:p%22%3C%3E%5C@"),
+ ],
+)
+def test_normalize_userinfo_encodes_component_data(userinfo, expected):
+ """Encode unsafe data while preserving userinfo delimiters and escaped
octets."""
+ assert normalize_userinfo(userinfo) == expected
+ assert (
+ url_normalize(f"http://{userinfo}example.com/";)
+ == f"http://{expected}example.com/";
+ )
+ assert normalize_userinfo(expected) == expected
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/tests/test_provide_url_scheme.py
new/url-normalize-3.0.1/tests/test_provide_url_scheme.py
--- old/url-normalize-3.0.0/tests/test_provide_url_scheme.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/tests/test_provide_url_scheme.py 2026-09-22
23:31:41.000000000 +0200
@@ -2,7 +2,7 @@
import pytest
-from url_normalize.url_normalize import provide_url_scheme
+from url_normalize.url_normalize import provide_url_scheme, url_normalize
@pytest.mark.parametrize(
@@ -30,3 +30,111 @@
actual = provide_url_scheme(url, default_scheme="http")
assert actual == expected
+
+
[email protected](
+ ("url", "expected"),
+ [
+ ("git+ssh://example.com/repo", "git+ssh://example.com/repo"),
+ ("postgresql://example.com/db", "postgresql://example.com/db"),
+ ("custom-long.scheme:opaque", "custom-long.scheme:opaque"),
+ ("TEL:123", "TEL:123"),
+ ("custom:123", "custom:123"),
+ ("//x:443/", "https://x:443/";),
+ ("//[::1]:443/", "https://[::1]:443/";),
+ ("x.co:8080/path", "https://x.co:8080/path";),
+ ("example.com:8080/path", "https://example.com:8080/path";),
+ ("localhost:8080/path", "https://localhost:8080/path";),
+ ("127.0.0.1:8080/", "https://127.0.0.1:8080/";),
+ ("[::1]:8080/", "https://[::1]:8080/";),
+ ("site/path:part", "https://site/path:part";),
+ ("1abc:thing", "https://1abc:thing";),
+ ("http:example.com", "http://example.com";),
+ ],
+)
+def test_provide_url_scheme_distinguishes_schemes_and_authorities(url,
expected):
+ """Recognize schemes of any length without confusing host ports or
paths."""
+ assert provide_url_scheme(url) == expected
+
+
[email protected]("prefix", [" ", "\t", "\r\n", "\u00a0", " \t\n"])
[email protected](
+ ("url", "expected"),
+ [
+ ("http://example.com/path ", "http://example.com/path "),
+ ("https://example.com/?q=x ", "https://example.com/?q=x "),
+ ("//x:443/path ", "https://x:443/path "),
+ ("example.com/path ", "https://example.com/path "),
+ ("git+ssh://example.com/repo ", "git+ssh://example.com/repo "),
+ ("mailto:[email protected]";, "mailto:[email protected]";),
+ ("/path ", "/path "),
+ ("-", "-"),
+ ("", ""),
+ ],
+)
+def test_provide_url_scheme_ignores_only_leading_whitespace(prefix, url,
expected):
+ """Remove leading whitespace before classifying the URL."""
+ assert provide_url_scheme(prefix + url) == expected
+
+
[email protected]("host", ["example.com", "localhost", "127.0.0.1"])
[email protected]("whitespace", [" ", "\u00a0", " \t\r\n"])
[email protected]("suffix", ["", "/path ", "?q=x ", "#part "])
+def test_provide_url_scheme_accepts_whitespace_after_bare_port(
+ host, whitespace, suffix
+):
+ """Recognize a bare authority with padding after its numeric port."""
+ value = f"{host}:443{whitespace}{suffix}"
+ cleaned = value.replace("\t", "").replace("\r", "").replace("\n", "")
+ assert provide_url_scheme(value) == f"https://{cleaned}";
+ path_suffix = suffix if suffix.startswith("/") else "/" + suffix
+ expected = f"https://{host}{path_suffix.replace(' ', '%20')}"
+ assert url_normalize(value) == expected
+ assert url_normalize(expected) == expected
+
+
[email protected]("control", ["\t", "\r", "\n"])
[email protected](
+ ("value", "expected"),
+ [
+ ("ht{control}tps://example.com/path", "https://example.com/path";),
+ ("https{control}://example.com/path", "https://example.com/path";),
+ ("git+{control}ssh://example.com/repo", "git+ssh://example.com/repo"),
+ ("mail{control}to:[email protected]", "mailto:[email protected]";),
+ ("/{control}/example.com/path", "https://example.com/path";),
+ ],
+)
+def test_scheme_detection_ignores_parser_control_characters(control, value,
expected):
+ """Keep the destination when urlsplit would discard tabs or line breaks."""
+ value = value.format(control=control)
+ assert provide_url_scheme(value) == expected
+ assert url_normalize(value, default_domain="default.example") == expected
+
+
[email protected]("prefix", ["\x00", "\x1f", "\x00\u00a0\x01 "])
[email protected](
+ ("value", "expected"),
+ [
+ ("http://example.com/path";, "http://example.com/path";),
+ ("//example.com/path", "https://example.com/path";),
+ ("/path", "/path"),
+ ("-", "-"),
+ ("", ""),
+ ],
+)
+def test_url_classification_ignores_leading_controls(prefix, value, expected):
+ """Remove leading controls before applying either default domain or
scheme."""
+ assert provide_url_scheme(prefix + value) == expected
+ assert url_normalize(prefix + value) == expected
+ with_domain = "https://default.example/path"; if value == "/path" else
expected
+ assert (
+ url_normalize(prefix + value, default_domain="default.example") ==
with_domain
+ )
+
+
+def test_scheme_cleanup_preserves_encoded_controls_and_trailing_spaces():
+ """Remove only raw parser controls, keeping escaped bytes and component
data."""
+ value = "ht\ntps://example.com/a%09b ?q=%0A%0D #part%00 "
+ assert url_normalize(value) == (
+ "https://example.com/a%09b%20?q=%0A%0D%20#part%00%20";
+ )
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/tests/test_url_humanize.py
new/url-normalize-3.0.1/tests/test_url_humanize.py
--- old/url-normalize-3.0.0/tests/test_url_humanize.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/tests/test_url_humanize.py 2026-09-22
23:31:41.000000000 +0200
@@ -101,12 +101,12 @@
assert package.url_humanize(value) == normalized
-def test_url_humanize_documents_encoded_slash_normalization() -> None:
- """Assert encoded slashes follow url_normalize path semantics."""
+def test_url_humanize_preserves_encoded_slash() -> None:
+ """Keep encoded slashes distinct from path segment boundaries."""
value = "https://example.com/a%2Fb";
humanized = package.url_humanize(value)
- assert humanized == "https://example.com/a/b";
+ assert humanized == value
assert package.url_normalize(humanized) == package.url_normalize(value)
@@ -120,3 +120,46 @@
"""Assert parts that break round-trip normalizations are kept as is."""
value = "https://example.com/a%3Fb";
assert package.url_humanize(value) == value
+
+
[email protected]("userinfo", ["user%5B", "user%5D", "user:%5B",
"user:%5D"])
+def test_url_humanize_rejects_candidates_that_raise(userinfo):
+ """Retain valid escaped credentials when decoding makes parsing fail."""
+ value = f"https://{userinfo}@example.com/";
+ assert package.url_humanize(value) == value
+ assert package.url_normalize(package.url_humanize(value)) == value
+
+
[email protected](
+ "component",
+ ["/{value}", "/?{value}=x", "/?q={value}", "/#{value}",
"//{value}@example.com/"],
+)
[email protected](
+ "escaped", ["%00", "%09", "%0A", "%0D", "%1B", "%7F", "%C2%85"]
+)
+def test_url_humanize_keeps_control_characters_encoded(component, escaped):
+ """Keep C0, DEL, and C1 controls encoded in every humanized component."""
+ suffix = component.format(value=f"a{escaped}b")
+ value = (
+ f"https:{suffix}" if suffix.startswith("//") else
f"https://example.com{suffix}";
+ )
+ assert package.url_humanize(value) == value
+
+
[email protected]("scheme", ["http", "https", "ftp"])
[email protected](
+ "suffix",
+ [
+ "//evil.example%[email protected]/",
+ "//name:evil.example%[email protected]/",
+ "//trusted.example/a%5Cb",
+ "//trusted.example/?a%5Cb=x",
+ "//trusted.example/?q=a%5Cb",
+ "//trusted.example/#a%5Cb",
+ ],
+)
+def test_url_humanize_keeps_backslashes_encoded(scheme, suffix):
+ """Keep backslashes encoded because browsers can treat them as
separators."""
+ value = f"{scheme}:{suffix}"
+ assert package.url_humanize(value) == value
+ assert package.url_normalize(package.url_humanize(value)) == value
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/tests/test_url_normalize.py
new/url-normalize-3.0.1/tests/test_url_normalize.py
--- old/url-normalize-3.0.0/tests/test_url_normalize.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/tests/test_url_normalize.py 2026-09-22
23:31:41.000000000 +0200
@@ -80,7 +80,7 @@
("http://example.com";, "http://example.com/";),
("http://example.com/?b&a";, "http://example.com/?b&a";),
("http://example.com/?q=%5c";, "http://example.com/?q=%5C";),
- ("http://example.com/?q=%C7";, "http://example.com/?q=%EF%BF%BD";),
+ ("http://example.com/?q=%C7";, "http://example.com/?q=%C7";),
("http://example.com/?q=C%CC%A7";, "http://example.com/?q=%C3%87";),
("http://EXAMPLE.COM/";, "http://example.com/";),
("http://example.com/%7Ejane";, "http://example.com/~jane";),
@@ -161,3 +161,47 @@
actual = url_normalize(url, default_scheme="http",
default_domain="example.com")
assert actual == expected
+
+
[email protected]("component", ["/{value}", "/?q={value}", "/#{value}"])
[email protected](
+ ("value", "expected"),
+ [
+ ("%ff", "%FF"),
+ ("%C7", "%C7"),
+ ("%E0%A4%A", "%E0%A4%25A"),
+ ("%FFe%CC%81", "%FF%C3%A9"),
+ ("e%CC%81", "%C3%A9"),
+ ("%EF%BF%BD", "%EF%BF%BD"),
+ ],
+)
+def test_url_normalize_preserves_invalid_utf8_octets(component, value,
expected):
+ """Preserve invalid octets while normalizing valid Unicode to NFC."""
+ prefix = "https://example.com";
+ normalized = prefix + component.format(value=expected)
+ assert url_normalize(prefix + component.format(value=value)) == normalized
+ assert url_normalize(normalized) == normalized
+
+
[email protected]("prefix", [" ", "\t", "\r\n", "\u00a0"])
[email protected]("default_domain", [None, "default.example"])
[email protected](
+ ("value", "expected"),
+ [
+ ("http://example.com/path ", "http://example.com/path%20";),
+ ("//x:443/path ", "https://x/path%20";),
+ ("example.com/?q=x ", "https://example.com/?q=x%20";),
+ ("/path ", "/path%20"),
+ ("", ""),
+ ("-", "-"),
+ ],
+)
+def test_url_normalize_strips_leading_whitespace_before_defaults(
+ prefix, default_domain, value, expected
+):
+ """Ignore leading whitespace before default-domain and scheme selection."""
+ if default_domain and value.startswith("/") and not value.startswith("//"):
+ expected = f"https://{default_domain}{expected}";
+ actual = url_normalize(prefix + value, default_domain=default_domain)
+ assert actual == expected
+ assert url_normalize(actual, default_domain=default_domain) == expected
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/url_normalize/__init__.py
new/url-normalize-3.0.1/url_normalize/__init__.py
--- old/url-normalize-3.0.0/url_normalize/__init__.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/url_normalize/__init__.py 2026-09-22
23:31:41.000000000 +0200
@@ -9,6 +9,6 @@
from .url_normalize import url_normalize
__license__ = "MIT"
-__version__ = "3.0.0"
+__version__ = "3.0.1"
__all__ = ["url_humanize", "url_normalize"]
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/url_normalize/cli.py
new/url-normalize-3.0.1/url_normalize/cli.py
--- old/url-normalize-3.0.0/url_normalize/cli.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/url_normalize/cli.py 2026-09-22
23:31:41.000000000 +0200
@@ -27,7 +27,9 @@
"-c",
"--charset",
default="utf-8",
- help="The charset of the URL. Default: utf-8",
+ help=(
+ "Retained for compatibility. Unicode characters use UTF-8 percent
encoding."
+ ),
)
parser.add_argument(
"-s",
@@ -39,7 +41,10 @@
"-f",
"--filter-params",
action="store_true",
- help="Filter common tracking parameters.",
+ help=(
+ "Keep only allowlisted query parameters. "
+ "Unknown domains have an empty default allowlist."
+ ),
)
parser.add_argument(
"-d",
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/url-normalize-3.0.0/url_normalize/generic_url_cleanup.py
new/url-normalize-3.0.1/url_normalize/generic_url_cleanup.py
--- old/url-normalize-3.0.0/url_normalize/generic_url_cleanup.py
2026-04-25 02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/url_normalize/generic_url_cleanup.py
2026-09-22 23:31:41.000000000 +0200
@@ -2,6 +2,8 @@
from __future__ import annotations
+from .tools import quote, unquote
+
def generic_url_cleanup(url: str) -> str:
"""Cleanup the URL from unnecessary data and convert to final form.
@@ -15,5 +17,11 @@
string : update url
"""
- url = url.replace("#!", "?_escaped_fragment_=")
- return url.rstrip("&? ")
+ base, fragment_separator, fragment = url.partition("#")
+ path, _, query = base.partition("?")
+ query = "&".join(param for param in query.split("&") if param)
+ if fragment.startswith("!"):
+ payload = quote(unquote(fragment[1:]), safe="/")
+ query += ("&" if query else "") + f"_escaped_fragment_={payload}"
+ fragment_separator = fragment = ""
+ return path + (f"?{query}" if query else "") + fragment_separator +
fragment
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/url_normalize/normalize_host.py
new/url-normalize-3.0.1/url_normalize/normalize_host.py
--- old/url-normalize-3.0.0/url_normalize/normalize_host.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/url_normalize/normalize_host.py 2026-09-22
23:31:41.000000000 +0200
@@ -24,6 +24,9 @@
"""
host = force_unicode(host, charset)
+ if host.startswith("[") and host.endswith("]"):
+ address, separator, zone = host.partition("%")
+ return address.lower() + separator + zone
host = host.lower()
host = host.strip(".")
@@ -31,8 +34,8 @@
parts = host.split(".")
try:
# Process each label separately to handle mixed unicode/ascii domains
- parts = [idna.encode(p, uts46=True).decode(charset) for p in parts if
p]
+ parts = [idna.encode(p, uts46=True).decode("ascii") for p in parts if
p]
return ".".join(parts)
except idna.IDNAError:
# Fallback to direct encoding if IDNA2008 processing fails
- return host.encode("idna").decode(charset)
+ return host.encode("idna").decode("ascii")
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/url_normalize/normalize_path.py
new/url-normalize-3.0.1/url_normalize/normalize_path.py
--- old/url-normalize-3.0.0/url_normalize/normalize_path.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/url_normalize/normalize_path.py 2026-09-22
23:31:41.000000000 +0200
@@ -2,7 +2,7 @@
from __future__ import annotations
-from .tools import quote, unquote
+from .tools import RESERVED_CHARACTERS, normalize_component
def normalize_path(path: str, scheme: str) -> str:
@@ -21,7 +21,7 @@
# Only perform percent-encoding where it is essential.
# Always use uppercase A-through-F characters when percent-encoding.
# All portions of the URI must be utf-8 encoded NFC from Unicode strings
- path = quote(unquote(path), "~:/#[]@!$&'()*+,;=")
+ path = normalize_component(path, "~:/[]@!$&'()*+,;=", RESERVED_CHARACTERS)
# Prevent dot-segments appearing in non-relative URI paths.
if scheme in {"", "http", "https", "ftp", "file"}:
output: list[str] = []
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/url_normalize/normalize_query.py
new/url-normalize-3.0.1/url_normalize/normalize_query.py
--- old/url-normalize-3.0.0/url_normalize/normalize_query.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/url_normalize/normalize_query.py 2026-09-22
23:31:41.000000000 +0200
@@ -3,7 +3,7 @@
from __future__ import annotations
from .param_allowlist import get_allowed_params
-from .tools import quote, unquote
+from .tools import normalize_component
QUERY_PARAM_SAFE_CHARS = "~:/?[]@!$'()*+,;"
@@ -23,7 +23,7 @@
"""
if not param:
return ""
- return quote(unquote(param), QUERY_PARAM_SAFE_CHARS)
+ return normalize_component(param, QUERY_PARAM_SAFE_CHARS, preserve="+")
def normalize_query(
@@ -48,17 +48,18 @@
if not query:
return ""
+ allowed_params = (
+ get_allowed_params(host, param_allowlist) if filter_params else None
+ )
processed = []
for param in query.split("&"):
if not param:
continue
- key, _, value = param.partition("=")
+ key, separator, value = param.partition("=")
key = process_query_param(key)
- if filter_params:
- allowed_params = get_allowed_params(host, param_allowlist)
- if key not in allowed_params:
- continue
+ if allowed_params is not None and key not in allowed_params:
+ continue
value = process_query_param(value)
- processed.append(f"{key}={value}" if value else key)
+ processed.append(f"{key}{separator}{value}")
return "&".join(processed)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/url-normalize-3.0.0/url_normalize/normalize_userinfo.py
new/url-normalize-3.0.1/url_normalize/normalize_userinfo.py
--- old/url-normalize-3.0.0/url_normalize/normalize_userinfo.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/url_normalize/normalize_userinfo.py 2026-09-22
23:31:41.000000000 +0200
@@ -2,6 +2,8 @@
from __future__ import annotations
+from .tools import RESERVED_CHARACTERS, normalize_component
+
def normalize_userinfo(userinfo: str) -> str:
"""Normalize userinfo part of the url.
@@ -15,4 +17,10 @@
"""
if userinfo in ["@", ":@"]:
return ""
- return userinfo
+ separator = "@" if userinfo.endswith("@") else ""
+ return (
+ normalize_component(
+ userinfo.removesuffix("@"), "!$&'()*+,;=:", RESERVED_CHARACTERS
+ )
+ + separator
+ )
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/url_normalize/param_allowlist.py
new/url-normalize-3.0.1/url_normalize/param_allowlist.py
--- old/url-normalize-3.0.0/url_normalize/param_allowlist.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/url_normalize/param_allowlist.py 2026-09-22
23:31:41.000000000 +0200
@@ -3,6 +3,8 @@
from __future__ import annotations
+from .normalize_host import normalize_host
+
DEFAULT_ALLOWLIST = {
"google.com": ["q", "ie"],
"baidu.com": ["wd", "ie"],
@@ -11,6 +13,15 @@
}
+def _normalize_domain(host: str) -> str:
+ """Canonicalize an allowlist domain without its port or www prefix."""
+ if host.startswith("[") and "]" in host:
+ host = host.partition("]")[0] + "]"
+ else:
+ host = host.partition(":")[0]
+ return normalize_host(host).removeprefix("www.")
+
+
def get_allowed_params(
host: str | None = None,
allowlist: dict | list | None = None,
@@ -35,14 +46,20 @@
if not host:
return set()
- # Normalize host by removing www and port
- domain = host.lower()
- domain = domain.removeprefix("www.")
- domain = domain.split(":")[0]
+ domain = _normalize_domain(host)
# Use default allowlist if none provided
if allowlist is None:
allowlist = DEFAULT_ALLOWLIST
- # Return allowed parameters for the domain, or an empty set if not found
- return set(allowlist.get(domain, []))
+ # Preserve exact canonical-key precedence, including an empty allowlist.
+ if domain in allowlist:
+ return set(allowlist[domain])
+ for key, params in allowlist.items():
+ try:
+ key_domain = _normalize_domain(key)
+ except UnicodeError:
+ continue
+ if key_domain == domain:
+ return set(params)
+ return set()
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/url-normalize-3.0.0/url_normalize/provide_url_scheme.py
new/url-normalize-3.0.1/url_normalize/provide_url_scheme.py
--- old/url-normalize-3.0.0/url_normalize/provide_url_scheme.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/url_normalize/provide_url_scheme.py 2026-09-22
23:31:41.000000000 +0200
@@ -2,7 +2,10 @@
from __future__ import annotations
+import re
+
from .normalize_scheme import DEFAULT_SCHEME
+from .tools import cleanup_url_input
# Schemes that require authority component reconstruction with //
AUTHORITY_SCHEMES = frozenset(["http", "https", "ftp", "ftps"])
@@ -19,14 +22,21 @@
string : updated url with validated/attached scheme
"""
- has_scheme = ":" in url[:7]
+ url = cleanup_url_input(url)
is_universal_scheme = url.startswith("//")
is_file_path = url == "-" or (url.startswith("/") and not
is_universal_scheme)
if not url or is_file_path:
return url
- if not has_scheme:
+ if is_universal_scheme:
+ return f"{default_scheme}:{url}"
+ if not re.match(r"[A-Za-z][A-Za-z0-9+.-]*:", url):
return f"{default_scheme}://{url.lstrip('/')}"
scheme_part, rest = url.split(":", 1)
+ # Dotted hosts and localhost followed by a numeric port are host inputs.
+ if ("." in scheme_part or scheme_part.lower() == "localhost") and re.match(
+ r"[0-9]+\s*(?:[/?#]|$)", rest
+ ):
+ return f"{default_scheme}://{url}"
if scheme_part.lower() not in AUTHORITY_SCHEMES:
# handle cases like tel:, mailto:, etc.
return url
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/url_normalize/tools.py
new/url-normalize-3.0.1/url_normalize/tools.py
--- old/url-normalize-3.0.0/url_normalize/tools.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/url_normalize/tools.py 2026-09-22
23:31:41.000000000 +0200
@@ -9,6 +9,15 @@
from urllib.parse import unquote as unquote_orig
from urllib.parse import urlsplit, urlunsplit
+RESERVED_CHARACTERS = ":/?#[]@!$&'()*+,;="
+
+
+def cleanup_url_input(url: str) -> str:
+ """Remove parser-ignored characters before URL classification."""
+ # Match urlsplit's C0/tab/newline cleanup, retaining Unicode whitespace
support.
+ url = re.sub(r"^[\x00-\x20\s]+", "", url)
+ return url.replace("\t", "").replace("\r", "").replace("\n", "")
+
class URL(NamedTuple):
"""URL components tuple.
@@ -36,9 +45,15 @@
URL
"""
- scheme, auth, path, query, fragment = urlsplit(url.strip())
- match = re.search(r"([^@]*@)?([^:]*):?(.*)", auth)
- (userinfo, host, port) = match.groups() # type: ignore # noqa: PGH003
+ scheme, auth, path, query, fragment = urlsplit(url)
+ userinfo, separator, host_port = auth.rstrip().rpartition("@")
+ userinfo += separator
+ if host_port.startswith("["):
+ host, _, port = host_port.partition("]")
+ host += "]"
+ port = port.removeprefix(":")
+ else:
+ host, _, port = host_port.partition(":")
return URL(
fragment=fragment,
host=host,
@@ -93,10 +108,9 @@
string : an unquoted and normalized string
"""
- string = unquote_orig(string)
+ string = unquote_orig(string, errors="surrogateescape")
string = force_unicode(string, charset)
- encoded_str = unicodedata.normalize("NFC", string).encode(charset)
- return encoded_str.decode(charset)
+ return unicodedata.normalize("NFC", string)
def quote(string: str, safe: str = "/") -> str:
@@ -110,4 +124,14 @@
string : quoted string
"""
- return quote_orig(string, safe)
+ return quote_orig(string, safe, errors="surrogateescape")
+
+
+def normalize_component(string: str, safe: str, preserve: str) -> str:
+ """Normalize escapes and Unicode without decoding protected delimiters."""
+ protected = "|".join(f"%{ord(character):02X}" for character in preserve)
+ parts = re.split(f"({protected})", string, flags=re.IGNORECASE)
+ return "".join(
+ part.upper() if index % 2 else quote(unquote(part), safe)
+ for index, part in enumerate(parts)
+ )
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/url_normalize/url_humanize.py
new/url-normalize-3.0.1/url_normalize/url_humanize.py
--- old/url-normalize-3.0.0/url_normalize/url_humanize.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/url_normalize/url_humanize.py 2026-09-22
23:31:41.000000000 +0200
@@ -2,6 +2,7 @@
from __future__ import annotations
+import unicodedata
from urllib.parse import unquote
import idna
@@ -27,14 +28,22 @@
def _replace_if_round_trips(url: URL, normalized: str, **changes: str) -> URL:
candidate = url._replace(**changes)
- if url_normalize(reconstruct_url(candidate)) == normalized:
- return candidate
+ try:
+ if url_normalize(reconstruct_url(candidate)) == normalized:
+ return candidate
+ except ValueError:
+ # Invalid candidate syntax or Unicode must not reject the original URL.
+ return url
return url
def _safe_unquote(value: str) -> str:
decoded = unquote(value)
- if UNICODE_REPLACEMENT_CHARACTER in decoded:
+ if (
+ UNICODE_REPLACEMENT_CHARACTER in decoded
+ or "\\" in decoded
+ or any(unicodedata.category(character) == "Cc" for character in
decoded)
+ ):
return value
return decoded
@@ -52,8 +61,10 @@
) -> tuple[URL, list[str]]:
candidate_parts = [*parts]
candidate_parts[index] = part
- candidate = url._replace(query="&".join(candidate_parts))
- if url_normalize(reconstruct_url(candidate)) == normalized:
+ candidate = _replace_if_round_trips(
+ url, normalized, query="&".join(candidate_parts)
+ )
+ if candidate is not url:
return candidate, candidate_parts
return url, parts
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/url_normalize/url_normalize.py
new/url-normalize-3.0.1/url_normalize/url_normalize.py
--- old/url-normalize-3.0.0/url_normalize/url_normalize.py 2026-04-25
02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/url_normalize/url_normalize.py 2026-09-22
23:31:41.000000000 +0200
@@ -18,7 +18,7 @@
from .normalize_userinfo import normalize_userinfo
from .provide_url_domain import provide_url_domain
from .provide_url_scheme import provide_url_scheme
-from .tools import deconstruct_url, reconstruct_url
+from .tools import cleanup_url_input, deconstruct_url, reconstruct_url
def url_normalize( # noqa: PLR0913
@@ -38,12 +38,12 @@
browsers handle data entered by the user:
>>> url_normalize('http://de.wikipedia.org/wiki/Elf (Begriffsklärung)')
- 'http://de.wikipedia.org/wiki/Elf%20%28Begriffskl%C3%A4rung%29'
+ 'http://de.wikipedia.org/wiki/Elf%20(Begriffskl%C3%A4rung)'
Params:
url : str | None : URL to normalize
charset : str : optional
- The target charset for the URL if the url was given as unicode
string
+ Retained for compatibility. Unicode characters use UTF-8 percent
encoding.
default_scheme : str : default scheme to use if none present
default_domain : str | None : optional
Default domain to use for absolute paths (starting with '/')
@@ -56,6 +56,8 @@
str | None : a normalized url
"""
+ if url is not None:
+ url = cleanup_url_input(url)
if not url:
return url
url = provide_url_domain(url, default_domain)
@@ -63,9 +65,11 @@
url = generic_url_cleanup(url)
url_elements = deconstruct_url(url)
url_elements = url_elements._replace(
+ host=normalize_host(url_elements.host, charset)
+ )
+ url_elements = url_elements._replace(
scheme=normalize_scheme(url_elements.scheme),
userinfo=normalize_userinfo(url_elements.userinfo),
- host=normalize_host(url_elements.host, charset),
query=normalize_query(
url_elements.query,
host=url_elements.host,
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/url-normalize-3.0.0/uv.lock
new/url-normalize-3.0.1/uv.lock
--- old/url-normalize-3.0.0/uv.lock 2026-04-25 02:23:36.000000000 +0200
+++ new/url-normalize-3.0.1/uv.lock 2026-09-22 23:31:41.000000000 +0200
@@ -632,7 +632,7 @@
[[package]]
name = "url-normalize"
-version = "3.0.0"
+version = "3.0.1"
source = { editable = "." }
dependencies = [
{ name = "idna" },
