Script 'mail_helper' called by obssrc
Hello community,
here is the log from the commit of package python-Protego for openSUSE:Factory
checked in at 2026-10-26 10:02:22
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comparing /work/SRC/openSUSE:Factory/python-Protego (Old)
and /work/SRC/openSUSE:Factory/.python-Protego.new.2252680 (New)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-Protego"
Mon Oct 26 10:02:22 2026 rev:11 rq:1392511 version:0.7.0
Changes:
--------
--- /work/SRC/openSUSE:Factory/python-Protego/python-Protego.changes
2026-07-07 21:04:03.796050809 +0200
+++
/work/SRC/openSUSE:Factory/.python-Protego.new.2252680/python-Protego.changes
2026-10-26 10:03:48.489290392 +0100
@@ -1,0 +2,46 @@
+Sun Oct 25 19:08:35 UTC 2026 - Dirk Müller <[email protected]>
+
+- update to 0.7.0:
+ * Backward-incompatible: Protego.parse() now raises a more
+ suitable TypeError instead of a ValueError when content is
+ not a string.
+ * Added official support for Python 3.15.
+ * can_fetch() now allows /robots.txt whatever the rules say, as
+ required by RFC 9309.
+ * A group now applies to a user agent only if its product token
+ appears in that user agent at a token boundary, so that User-
+ agent: bot no longer applies to mybot. A product token within
+ a whole User-Agent header value, e.g. Mozilla/5.0
+ (compatible; mybot/1.0), still matches.
+ * Percent-encoding is now normalized in the query string and
+ the parameters of a URL, and not only in its path, so that a
+ rule matches whichever spelling either of them uses, e.g.
+ Disallow: /a?b=ツ now matches /a?b=%E3%83%84. The fragment is
+ now left out of matching altogether.
+ * Visit-time now also accepts a hyphen between the two times,
+ e.g. 0400-0845, and times with a single-digit hour, e.g. 400.
+ * Malformed Crawl-delay values, such as a negative or infinite
+ number, and Request-rate values with zero requests or zero
+ seconds, are now ignored, as other malformed values already
+ were.
+ * Improved the handling of invalid and misspelled lines.
+ Sitemap and Host lines are now read anywhere in the file,
+ including before any User-agent line, and no longer merge the
+ groups they sit between. A User-agent line without a value
+ still ends the preceding group. Lines written without a colon
+ are now salvaged for every directive, Visit-time included,
+ and with any whitespace as the separator. A line whose field
+ is not a known directive is no longer read as one.
+ * Fixed matching of URLs whose path contains =, which no rule
+ could match because it was percent-encoded in the URL but not
+ in the rule.
+ * Fixed matching of URLs whose path starts with //, which got
+ extra slashes before being matched.
+ * Fixed Allow: …/index.html, which, besides allowing the parent
+ directory as intended, also allowed URLs with a $ right after
+ that directory.
+ * Improved matching and parsing performance.
+ * Documentation improvements, including a rewritten parser
+ comparison table generated from benchmarks.
+
+-------------------------------------------------------------------
Old:
----
protego-0.6.2.tar.gz
New:
----
protego-0.7.0.tar.gz
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other differences:
------------------
++++++ python-Protego.spec ++++++
--- /var/tmp/diff_new_pack.KEb6bm/_old 2026-10-26 10:03:49.359326879 +0100
+++ /var/tmp/diff_new_pack.KEb6bm/_new 2026-10-26 10:03:49.361326963 +0100
@@ -18,7 +18,7 @@
%{?sle15_python_module_pythons}
Name: python-Protego
-Version: 0.6.2
+Version: 0.7.0
Release: 0
Summary: Pure-Python robotstxt parser with support for modern
conventions
License: BSD-3-Clause
++++++ protego-0.6.2.tar.gz -> protego-0.7.0.tar.gz ++++++
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/protego-0.6.2/.gitignore new/protego-0.7.0/.gitignore
--- old/protego-0.6.2/.gitignore 2020-02-02 01:00:00.000000000 +0100
+++ new/protego-0.7.0/.gitignore 2020-02-02 01:00:00.000000000 +0100
@@ -21,3 +21,6 @@
.coverage
.coverage.*
coverage.*
+
+# test reports
+testenv.junit.xml
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/protego-0.6.2/CHANGELOG.rst
new/protego-0.7.0/CHANGELOG.rst
--- old/protego-0.6.2/CHANGELOG.rst 2020-02-02 01:00:00.000000000 +0100
+++ new/protego-0.7.0/CHANGELOG.rst 2020-02-02 01:00:00.000000000 +0100
@@ -2,6 +2,59 @@
Protego changelog
=================
+0.7.0 (2026-09-21)
+==================
+
+- **Backward-incompatible:** ``Protego.parse()`` now raises a more suitable
+ ``TypeError`` instead of a ``ValueError`` when ``content`` is not a string.
+
+- Added official support for Python 3.15.
+
+- ``can_fetch()`` now allows ``/robots.txt`` whatever the rules say, as
+ required by RFC 9309.
+
+- A group now applies to a user agent only if its product token appears in
+ that user agent at a token boundary, so that ``User-agent: bot`` no longer
+ applies to ``mybot``. A product token within a whole ``User-Agent`` header
+ value, e.g. ``Mozilla/5.0 (compatible; mybot/1.0)``, still matches.
+
+- Percent-encoding is now normalized in the query string and the parameters
+ of a URL, and not only in its path, so that a rule matches whichever
+ spelling either of them uses, e.g. ``Disallow: /a?b=ツ`` now matches
+ ``/a?b=%E3%83%84``. The fragment is now left out of matching altogether.
+
+- ``Visit-time`` now also accepts a hyphen between the two times, e.g.
+ ``0400-0845``, and times with a single-digit hour, e.g. ``400``.
+
+- Malformed ``Crawl-delay`` values, such as a negative or infinite number,
+ and ``Request-rate`` values with zero requests or zero seconds, are now
+ ignored, as other malformed values already were.
+
+- Improved the handling of invalid and misspelled lines.
+
+ ``Sitemap`` and ``Host`` lines are now read anywhere in the file,
+ including before any ``User-agent`` line, and no longer merge the groups
+ they sit between. A ``User-agent`` line without a value still ends the
+ preceding group. Lines written without a colon are now salvaged for every
+ directive, ``Visit-time`` included, and with any whitespace as the
+ separator. A line whose field is not a known directive is no longer read
+ as one.
+
+- Fixed matching of URLs whose path contains ``=``, which no rule could
+ match because it was percent-encoded in the URL but not in the rule.
+
+- Fixed matching of URLs whose path starts with ``//``, which got extra
+ slashes before being matched.
+
+- Fixed ``Allow: …/index.html``, which, besides allowing the parent
+ directory as intended, also allowed URLs with a ``$`` right after that
+ directory.
+
+- Improved matching and parsing performance.
+
+- Documentation improvements, including a rewritten parser comparison table
+ generated from benchmarks.
+
0.6.2 (2026-06-25)
==================
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/protego-0.6.2/PKG-INFO new/protego-0.7.0/PKG-INFO
--- old/protego-0.6.2/PKG-INFO 2020-02-02 01:00:00.000000000 +0100
+++ new/protego-0.7.0/PKG-INFO 2020-02-02 01:00:00.000000000 +0100
@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
Name: Protego
-Version: 0.6.2
+Version: 0.7.0
Summary: Pure-Python robots.txt parser with support for modern conventions
Project-URL: Homepage, https://github.com/scrapy/protego
Project-URL: Source, https://github.com/scrapy/protego
@@ -20,6 +20,7 @@
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Internet :: WWW/HTTP
@@ -39,8 +40,15 @@
:target:
https://github.com/scrapy/protego/actions/workflows/tests-ubuntu.yml
:alt: CI
-Protego is a pure-Python ``robots.txt`` parser with support for modern
-conventions.
+Protego is a pure-Python ``robots.txt`` parser. It implements the parsing and
+URL matching rules of `RFC 9309`_, and additionally supports the
+``Crawl-delay``, ``Request-rate``, ``Visit-time`` and ``Host`` extensions.
+
+Fetching ``robots.txt`` is up to you, and so are the parts of `RFC 9309`_ that
+govern it, such as the handling of HTTP status codes and redirects, caching,
and
+imposing a parsing limit.
+
+.. _RFC 9309: https://www.rfc-editor.org/rfc/rfc9309.html
Install
@@ -115,27 +123,95 @@
==========
The following table compares Protego to the most popular ``robots.txt`` parsers
-implemented in Python or featuring Python bindings:
-
-+----------------------------+---------+-----------------+--------+---------------------------+
-| | Protego | RobotFileParser | Reppy |
Robotexclusionrulesparser |
-+============================+=========+=================+========+===========================+
-| Implementation language | Python | Python | C++ | Python
|
-+----------------------------+---------+-----------------+--------+---------------------------+
-| Reference specification | Google_ | `Martijn Koster’s 1996 draft`_
|
-+----------------------------+---------+-----------------+--------+---------------------------+
-| `Wildcard support`_ | ✓ | | ✓ | ✓
|
-+----------------------------+---------+-----------------+--------+---------------------------+
-| `Length-based precedence`_ | ✓ | | ✓ |
|
-+----------------------------+---------+-----------------+--------+---------------------------+
-| Performance_ | | +40% | +1300% | -25%
|
-+----------------------------+---------+-----------------+--------+---------------------------+
-
-.. _Google: https://developers.google.com/search/reference/robots_txt
-.. _Length-based precedence:
https://developers.google.com/search/reference/robots_txt#order-of-precedence-for-group-member-lines
-.. _Martijn Koster’s 1996 draft: https://www.robotstxt.org/norobots-rfc.txt
-.. _Performance: https://anubhavp28.github.io/gsoc-weekly-checkin-12/
-.. _Wildcard support:
https://developers.google.com/search/reference/robots_txt#url-matching-based-on-path-values
+implemented in Python. Performance is the speed difference against Protego, so
+a positive value means faster than Protego. It is measured over the
+``robots.txt`` of 100 of the most visited websites: the time taken to check the
+URLs their homepages link to, and the time taken to parse the files themselves.
+
+.. The table below is generated by benchmarks/compare.py. Regenerate it
+ rather than editing it by hand.
+
+ Its feature rows are probe results against the exact releases named in
+ "Version tested", not claims about the parsers in general. RobotFileParser
+ is the one that moves: CPython taught urllib.robotparser the RFC 9309
+ wildcards and longest-match precedence during the 3.14 series, after
+ 3.14.3. Generated on 3.14.3 or older, its wildcard and length-based
+ precedence cells come out empty; generated on a later 3.14 they carry a
+ checkmark. Both are what the probes measured on that interpreter, so do not
+ flip those two cells by hand in either direction.
+
+.. comparison-table-start
+
+.. list-table::
+ :header-rows: 1
+ :stub-columns: 1
+
+ * -
+ - Protego
+ - RobotFileParser
+ - robotspy
+ - Robotexclusionrulesparser
+ * - Version tested
+ -
+ - Python 3.14.7
+ - 0.13.0
+ - 1.7.1
+ * - Reference specification
+ - `RFC 9309`_
+ - `Martijn Koster's 1996 draft`_
+ - `RFC 9309`_
+ - `Martijn Koster's 1996 draft`_
+ * - `Wildcard support`_
+ - ✓
+ - ✓
+ - ✓
+ - ✓
+ * - `Length-based precedence`_
+ - ✓
+ - ✓
+ - ✓
+ -
+ * - Crawl-delay
+ - ✓
+ - ✓
+ -
+ -
+ * - Request-rate
+ - ✓
+ -
+ -
+ -
+ * - Visit-time
+ - ✓
+ -
+ -
+ -
+ * - Sitemaps
+ - ✓
+ - ✓
+ - ✓
+ - ✓
+ * - Host
+ - ✓
+ -
+ -
+ -
+ * - Matching performance
+ -
+ - -62%
+ - -74%
+ - -96%
+ * - Parsing performance
+ -
+ - -71%
+ - +39%
+ - +56%
+
+.. comparison-table-end
+
+.. _Length-based precedence:
https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2
+.. _Martijn Koster's 1996 draft: https://www.robotstxt.org/norobots-rfc.txt
+.. _Wildcard support: https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.3
API Reference
@@ -161,6 +237,11 @@
* ``can_fetch(url, user_agent)`` Return True if the user agent can fetch the
URL, otherwise return ``False``.
+ *user_agent* may be a product token, such as ``"mybot"``, or a whole
+ ``User-Agent`` header value, such as ``"Mozilla/5.0 (compatible;
+ mybot/1.0)"``; a group applies when its product token appears in
+ *user_agent* at a token boundary.
+
* ``crawl_delay(user_agent)`` Return the crawl delay specified for the user
agent as a float. If nothing is specified, return ``None``.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/protego-0.6.2/README.rst new/protego-0.7.0/README.rst
--- old/protego-0.6.2/README.rst 2020-02-02 01:00:00.000000000 +0100
+++ new/protego-0.7.0/README.rst 2020-02-02 01:00:00.000000000 +0100
@@ -10,8 +10,15 @@
:target:
https://github.com/scrapy/protego/actions/workflows/tests-ubuntu.yml
:alt: CI
-Protego is a pure-Python ``robots.txt`` parser with support for modern
-conventions.
+Protego is a pure-Python ``robots.txt`` parser. It implements the parsing and
+URL matching rules of `RFC 9309`_, and additionally supports the
+``Crawl-delay``, ``Request-rate``, ``Visit-time`` and ``Host`` extensions.
+
+Fetching ``robots.txt`` is up to you, and so are the parts of `RFC 9309`_ that
+govern it, such as the handling of HTTP status codes and redirects, caching,
and
+imposing a parsing limit.
+
+.. _RFC 9309: https://www.rfc-editor.org/rfc/rfc9309.html
Install
@@ -86,27 +93,95 @@
==========
The following table compares Protego to the most popular ``robots.txt`` parsers
-implemented in Python or featuring Python bindings:
-
-+----------------------------+---------+-----------------+--------+---------------------------+
-| | Protego | RobotFileParser | Reppy |
Robotexclusionrulesparser |
-+============================+=========+=================+========+===========================+
-| Implementation language | Python | Python | C++ | Python
|
-+----------------------------+---------+-----------------+--------+---------------------------+
-| Reference specification | Google_ | `Martijn Koster’s 1996 draft`_
|
-+----------------------------+---------+-----------------+--------+---------------------------+
-| `Wildcard support`_ | ✓ | | ✓ | ✓
|
-+----------------------------+---------+-----------------+--------+---------------------------+
-| `Length-based precedence`_ | ✓ | | ✓ |
|
-+----------------------------+---------+-----------------+--------+---------------------------+
-| Performance_ | | +40% | +1300% | -25%
|
-+----------------------------+---------+-----------------+--------+---------------------------+
-
-.. _Google: https://developers.google.com/search/reference/robots_txt
-.. _Length-based precedence:
https://developers.google.com/search/reference/robots_txt#order-of-precedence-for-group-member-lines
-.. _Martijn Koster’s 1996 draft: https://www.robotstxt.org/norobots-rfc.txt
-.. _Performance: https://anubhavp28.github.io/gsoc-weekly-checkin-12/
-.. _Wildcard support:
https://developers.google.com/search/reference/robots_txt#url-matching-based-on-path-values
+implemented in Python. Performance is the speed difference against Protego, so
+a positive value means faster than Protego. It is measured over the
+``robots.txt`` of 100 of the most visited websites: the time taken to check the
+URLs their homepages link to, and the time taken to parse the files themselves.
+
+.. The table below is generated by benchmarks/compare.py. Regenerate it
+ rather than editing it by hand.
+
+ Its feature rows are probe results against the exact releases named in
+ "Version tested", not claims about the parsers in general. RobotFileParser
+ is the one that moves: CPython taught urllib.robotparser the RFC 9309
+ wildcards and longest-match precedence during the 3.14 series, after
+ 3.14.3. Generated on 3.14.3 or older, its wildcard and length-based
+ precedence cells come out empty; generated on a later 3.14 they carry a
+ checkmark. Both are what the probes measured on that interpreter, so do not
+ flip those two cells by hand in either direction.
+
+.. comparison-table-start
+
+.. list-table::
+ :header-rows: 1
+ :stub-columns: 1
+
+ * -
+ - Protego
+ - RobotFileParser
+ - robotspy
+ - Robotexclusionrulesparser
+ * - Version tested
+ -
+ - Python 3.14.7
+ - 0.13.0
+ - 1.7.1
+ * - Reference specification
+ - `RFC 9309`_
+ - `Martijn Koster's 1996 draft`_
+ - `RFC 9309`_
+ - `Martijn Koster's 1996 draft`_
+ * - `Wildcard support`_
+ - ✓
+ - ✓
+ - ✓
+ - ✓
+ * - `Length-based precedence`_
+ - ✓
+ - ✓
+ - ✓
+ -
+ * - Crawl-delay
+ - ✓
+ - ✓
+ -
+ -
+ * - Request-rate
+ - ✓
+ -
+ -
+ -
+ * - Visit-time
+ - ✓
+ -
+ -
+ -
+ * - Sitemaps
+ - ✓
+ - ✓
+ - ✓
+ - ✓
+ * - Host
+ - ✓
+ -
+ -
+ -
+ * - Matching performance
+ -
+ - -62%
+ - -74%
+ - -96%
+ * - Parsing performance
+ -
+ - -71%
+ - +39%
+ - +56%
+
+.. comparison-table-end
+
+.. _Length-based precedence:
https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2
+.. _Martijn Koster's 1996 draft: https://www.robotstxt.org/norobots-rfc.txt
+.. _Wildcard support: https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.3
API Reference
@@ -132,6 +207,11 @@
* ``can_fetch(url, user_agent)`` Return True if the user agent can fetch the
URL, otherwise return ``False``.
+ *user_agent* may be a product token, such as ``"mybot"``, or a whole
+ ``User-Agent`` header value, such as ``"Mozilla/5.0 (compatible;
+ mybot/1.0)"``; a group applies when its product token appears in
+ *user_agent* at a token boundary.
+
* ``crawl_delay(user_agent)`` Return the crawl delay specified for the user
agent as a float. If nothing is specified, return ``None``.
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/protego-0.6.2/pyproject.toml
new/protego-0.7.0/pyproject.toml
--- old/protego-0.6.2/pyproject.toml 2020-02-02 01:00:00.000000000 +0100
+++ new/protego-0.7.0/pyproject.toml 2020-02-02 01:00:00.000000000 +0100
@@ -4,7 +4,7 @@
[project]
name = "Protego"
-version = "0.6.2"
+version = "0.7.0"
description = "Pure-Python robots.txt parser with support for modern
conventions"
authors = [
{ name = "Anubhav Patel", email = "[email protected]" },
@@ -20,6 +20,7 @@
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
+ "Programming Language :: Python :: 3.15",
"Programming Language :: Python :: Implementation :: CPython",
"Programming Language :: Python :: Implementation :: PyPy",
"Topic :: Internet :: WWW/HTTP",
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/protego-0.6.2/src/protego/_protego.py
new/protego-0.7.0/src/protego/_protego.py
--- old/protego-0.6.2/src/protego/_protego.py 2020-02-02 01:00:00.000000000
+0100
+++ new/protego-0.7.0/src/protego/_protego.py 2020-02-02 01:00:00.000000000
+0100
@@ -2,6 +2,7 @@
import logging
from typing import TYPE_CHECKING
+from urllib.parse import urlparse
from ._ruleset import RequestRate, VisitTime, _RuleSet
@@ -11,6 +12,8 @@
# typing.Self requires Python 3.11
from typing_extensions import Self
+ from ._urlpattern import _URLPattern
+
logger = logging.getLogger(__name__)
@@ -31,19 +34,71 @@
_VISIT_TIME_DIRECTIVE = {"visit-time", "visit time"}
_HOST_DIRECTIVE = {"host"}
+# Directives that apply to the whole site rather than to a record group.
+_SITE_WIDE_DIRECTIVES = _SITEMAP_DIRECTIVE | _HOST_DIRECTIVE
+
+_ALL_DIRECTIVES = (
+ _DISALLOW_DIRECTIVE
+ | _ALLOW_DIRECTIVE
+ | _USER_AGENT_DIRECTIVE
+ | _SITEMAP_DIRECTIVE
+ | _CRAWL_DELAY_DIRECTIVE
+ | _REQUEST_RATE_DIRECTIVE
+ | _VISIT_TIME_DIRECTIVE
+ | _HOST_DIRECTIVE
+)
+
+_MAX_DIRECTIVE_WORDS = max(len(alias.split()) for alias in _ALL_DIRECTIVES)
+
def _is_valid_directive_field(field: str) -> bool:
- return any(
- [
- field in _DISALLOW_DIRECTIVE,
- field in _ALLOW_DIRECTIVE,
- field in _USER_AGENT_DIRECTIVE,
- field in _SITEMAP_DIRECTIVE,
- field in _CRAWL_DELAY_DIRECTIVE,
- field in _REQUEST_RATE_DIRECTIVE,
- field in _HOST_DIRECTIVE,
- ]
- )
+ return field in _ALL_DIRECTIVES
+
+
+def _extract_directive(line: str) -> tuple[str, str] | None:
+ """Split a non-empty line into a lowercased directive field and a
+ stripped value, or return None if the line cannot be read as a
+ directive. A line with "<field>:<value>" shape is returned even when
+ the field is not a known directive, so that the caller can count it as
+ an invalid directive.
+
+ We are more generous than the specs: we allow e.g. "User-agent: foo",
+ "User-agent foo", "User agent: foo" and "User agent foo" even though all
+ specs only specify the first form. We also tolerate some misspellings.
+ """
+ field: str | None = None
+ value = ""
+ if ":" in line:
+ field, raw_value = line.split(":", 1)
+ field = field.strip().lower()
+ value = raw_value.strip()
+ if _is_valid_directive_field(field):
+ # Happy path
+ return field, value
+ if not raw_value or raw_value[0].isspace():
+ # The field is not a known directive, but the colon reads as a
+ # field separator (it ends the line or is followed by
+ # whitespace), so we keep the "<field>:<value>" interpretation
+ # and treat the line as an unknown directive.
+ return field, value
+
+ # There is no colon separator, or the field found is not a known
+ # directive and its colon does not read as a field separator. We will
+ # be generous here and give the line a second chance, parsing it as a
+ # whitespace-separated "<field> <value>" directive. This also salvages
+ # lines whose value contains a colon of its own, e.g.
+ # "Sitemap https://example.com/sitemap.xml";.
+ words = line.split()
+ for word_count in range(1, _MAX_DIRECTIVE_WORDS + 1):
+ # Trying "<multiword> <field> <value>"
+ candidate = " ".join(words[:word_count]).lower()
+ if _is_valid_directive_field(candidate):
+ return candidate, " ".join(words[word_count:])
+
+ if field is None:
+ return None
+ # Salvage failed; keep the unknown "<field>:<value>" directive.
+ return field, value
class Protego:
@@ -60,6 +115,9 @@
# A memoization table mapping user agents (used in queries) to matched
rule sets.
self._matched_rule_set: dict[str, _RuleSet | None] = {}
+ # URL patterns by directive value, shared across rule sets.
+ self._url_patterns: dict[str, _URLPattern | None] = {}
+
self._total_line_seen = 0
self._invalid_directive_seen = 0
self._total_directive_seen = 0
@@ -68,7 +126,7 @@
def parse(cls, content: str) -> Self:
o = cls()
if not isinstance(content, str):
- raise ValueError(f"Protego.parse expects str, got
{type(content).__name__}")
+ raise TypeError(f"Protego.parse expects str, got
{type(content).__name__}")
o._parse_robotstxt(content)
return o
@@ -95,37 +153,35 @@
if not line:
continue
- # Format for a valid robots.txt rule is "<field>:<value>"
- if line.find(":") != -1:
- field, value = line.split(":", 1)
- else:
- # We will be generous here and give it a second chance.
- parts = line.split(" ")
- if len(parts) < 2:
- continue
-
- possible_filed = parts[0]
- for i in range(1, len(parts)):
- if _is_valid_directive_field(possible_filed):
- field, value = possible_filed, " ".join(parts[i:])
- break
- possible_filed += " " + parts[i]
- else:
- continue
+ parsed = _extract_directive(line)
+ if parsed is None:
+ continue
+ field, value = parsed
- field = field.strip().lower()
- value = value.strip()
+ # A user agent line, even one without a usable value, ends the
+ # previous record group.
+ if (
+ field in _USER_AGENT_DIRECTIVE
+ and previous_rule_field is not None
+ and previous_rule_field not in _USER_AGENT_DIRECTIVE
+ ):
+ current_rule_sets = []
+
+ # Sitemap and Host are site-wide directives and don't matter for
+ # record grouping.
+ if field not in _SITE_WIDE_DIRECTIVES:
+ previous_rule_field = field
# Ignore rules with no value part (e.g. "Disallow: ", "Allow: ").
if not value:
- previous_rule_field = field
continue
- # Ignore rules without a corresponding user agent.
+ # Ignore rules without a corresponding user agent, except
+ # site-wide directives which need none.
if (
not current_rule_sets
and field not in _USER_AGENT_DIRECTIVE
- and field not in _SITEMAP_DIRECTIVE
+ and field not in _SITE_WIDE_DIRECTIVES
):
logger.debug(
f"Rule at line {self._total_line_seen} without any user
agent to enforce it on."
@@ -135,12 +191,6 @@
self._total_directive_seen += 1
if field in _USER_AGENT_DIRECTIVE:
- if (
- previous_rule_field
- and previous_rule_field not in _USER_AGENT_DIRECTIVE
- ):
- current_rule_sets = []
-
# Wildcards are not supported in the user agent values.
# We will be generous here and remove all the wildcards.
user_agent = value.strip().lower()
@@ -192,8 +242,6 @@
else:
self._invalid_directive_seen += 1
- previous_rule_field = field
-
for rule_set in self._user_agents.values():
rule_set.finalize_rules()
@@ -217,6 +265,8 @@
def can_fetch(self, url: str, user_agent: str) -> bool:
"""Return True if the user agent can fetch the URL, otherwise return
False."""
+ if "/robots.txt" in url and urlparse(url).path == "/robots.txt":
+ return True
matched_rule_set = self._get_matching_rule_set(user_agent)
if not matched_rule_set:
return True
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/protego-0.6.2/src/protego/_ruleset.py
new/protego-0.7.0/src/protego/_ruleset.py
--- old/protego-0.6.2/src/protego/_ruleset.py 2020-02-02 01:00:00.000000000
+0100
+++ new/protego-0.7.0/src/protego/_ruleset.py 2020-02-02 01:00:00.000000000
+0100
@@ -1,6 +1,7 @@
from __future__ import annotations
import logging
+import math
from typing import TYPE_CHECKING, NamedTuple
from ._urlpattern import _URLPattern
@@ -38,6 +39,8 @@
def __init__(self, parser_instance: Protego):
self.user_agent: str | None = None
self._rules: list[_Rule] = []
+ self._plain_prefixes: tuple[str, ...] | None = None
+ self._special_rules: list[_Rule] = []
self._crawl_delay: float | None = None
self._req_rate: RequestRate | None = None
self._visit_time: VisitTime | None = None
@@ -49,42 +52,95 @@
robotname = robotname.strip().lower()
if self.user_agent == "*":
return 1
- if self.user_agent in robotname:
- return len(self.user_agent)
+ # Match the product token only at a token boundary. This avoids
+ # false positives such as "bot" matching "mybot", while still
+ # matching a token within a full User-Agent header, e.g.
+ # "Mozilla/5.0 (compatible; Foobot/1.0)" matching "foobot".
+ index = robotname.find(self.user_agent)
+ while index != -1:
+ if index == 0 or not (
+ robotname[index - 1].isalnum() or robotname[index - 1] in "-_"
+ ):
+ return len(self.user_agent)
+ index = robotname.find(self.user_agent, index + 1)
return 0
+ def _url_pattern(self, pattern: str) -> _URLPattern | None:
+ """Return the URL pattern for a directive value, or None if the value
+ quotes to nothing.
+
+ The same value usually appears once per user agent in a robots.txt,
+ so patterns are shared across the rule sets of a parser.
+ """
+ cache = self._parser_instance._url_patterns
+ if pattern not in cache:
+ quoted = _quote_pattern(pattern)
+ cache[pattern] = _URLPattern(quoted) if quoted else None
+ return cache[pattern]
+
def allow(self, pattern: str) -> None:
if "$" in pattern:
self.allow(pattern.replace("$", _hexescape("$")))
- pattern = _quote_pattern(pattern)
- if not pattern:
+ url_pattern = self._url_pattern(pattern)
+ if url_pattern is None:
return
- self._rules.append(_Rule(field="allow", value=_URLPattern(pattern)))
+ self._rules.append(_Rule(field="allow", value=url_pattern))
# If index.html is allowed, we interpret this as / being allowed too.
- if pattern.endswith("/index.html"):
- self.allow(pattern[:-10] + "$")
+ page = "index.html"
+ quoted = url_pattern._pattern
+ if quoted.endswith(f"/{page}"):
+ # Add the rule directly; going through allow() would treat the
+ # "$" anchor as a literal dollar sign too.
+ self._rules.append(
+ _Rule(field="allow",
value=_URLPattern(quoted.removesuffix(page) + "$"))
+ )
def disallow(self, pattern: str) -> None:
if "$" in pattern:
self.disallow(pattern.replace("$", _hexescape("$")))
- pattern = _quote_pattern(pattern)
- if not pattern:
+ url_pattern = self._url_pattern(pattern)
+ if url_pattern is None:
return
- self._rules.append(_Rule(field="disallow", value=_URLPattern(pattern)))
+ self._rules.append(_Rule(field="disallow", value=url_pattern))
def finalize_rules(self) -> None:
self._rules.sort(
key=lambda r: (r.value.priority, r.field == "allow"), reverse=True
)
+ def _build_index(self) -> tuple[str, ...]:
+ """Split the rules into the patterns that can only match as a prefix
+ of a URL and the rules that can match otherwise.
+
+ Built on the first match rather than at parse time because a
+ robots.txt declares many rule sets and a crawler queries one.
+ """
+ prefixes = []
+ special = []
+ for rule in self._rules:
+ pattern = rule.value
+ if pattern._contains_asterisk or pattern._contains_dollar:
+ special.append(rule)
+ else:
+ prefixes.append(pattern._pattern)
+ self._special_rules = special
+ self._plain_prefixes = tuple(prefixes)
+ return self._plain_prefixes
+
def can_fetch(self, url: str) -> bool:
"""Return if the url can be fetched."""
url = _quote_path(url)
+ # A plain pattern matches only as a prefix of the URL, so a single
+ # startswith over all of them rules every one of them out at once.
+ prefixes = self._plain_prefixes
+ if prefixes is None:
+ prefixes = self._build_index()
+ rules = self._rules if url.startswith(prefixes) else
self._special_rules
allowed = True
- for rule in self._rules:
+ for rule in rules:
if rule.value.match(url):
if rule.field == "disallow":
allowed = False
@@ -99,13 +155,17 @@
@crawl_delay.setter
def crawl_delay(self, delay: str) -> None:
try:
- self._crawl_delay = float(delay)
+ parsed_delay = float(delay)
except ValueError:
+ parsed_delay = None
+ if parsed_delay is None or not math.isfinite(parsed_delay) or
parsed_delay < 0:
# Value is malformed, do nothing.
logger.debug(
f"Malformed rule at line
{self._parser_instance._total_line_seen} : "
f"cannot set crawl delay to '{delay}'. Ignoring this rule."
)
+ return
+ self._crawl_delay = parsed_delay
@property
def request_rate(self) -> RequestRate | None:
@@ -130,6 +190,9 @@
seconds = int(seconds_str)
requests = int(requests_str)
+ if requests <= 0 or seconds <= 0:
+ raise ValueError(f"Request rate must be positive: {value!r}")
+
if time_unit == "m":
seconds *= 60
elif time_unit == "h":
@@ -141,7 +204,7 @@
end_time = None
if time_period:
start_time, end_time = _parse_time_period(time_period)
- except Exception:
+ except Exception: # noqa: BLE001
# Value is malformed, do nothing.
logger.debug(
f"Malformed rule at line
{self._parser_instance._total_line_seen} : "
@@ -159,8 +222,13 @@
@visit_time.setter
def visit_time(self, value: str) -> None:
try:
- start_time, end_time = _parse_time_period(value, separator=" ")
- except Exception:
+ try:
+ # "0400-0845"
+ start_time, end_time = _parse_time_period(value)
+ except ValueError:
+ # "0400 0845"
+ start_time, end_time = _parse_time_period(value, separator=" ")
+ except Exception: # noqa: BLE001
logger.debug(
f"Malformed rule at line
{self._parser_instance._total_line_seen} : "
f"cannot set visit time using '{value}'. Ignoring this rule."
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/protego-0.6.2/src/protego/_utils.py
new/protego-0.7.0/src/protego/_utils.py
--- old/protego-0.6.2/src/protego/_utils.py 2020-02-02 01:00:00.000000000
+0100
+++ new/protego-0.7.0/src/protego/_utils.py 2020-02-02 01:00:00.000000000
+0100
@@ -1,17 +1,47 @@
from __future__ import annotations
+import re
from datetime import time
-from urllib.parse import ParseResult, quote, urlparse, urlunparse
+from urllib.parse import quote, urlsplit
_HEX_DIGITS = set("0123456789ABCDEFabcdef")
+# Escapes left as they are while normalizing a URL for comparison: the path
+# separator, and "%" so that escapes already present are not doubled.
+_KEPT = "/%"
+
+# Patterns additionally keep "*", which stands for any sequence of characters.
+_PATTERN_KEPT = f"{_KEPT}*"
+
+# Characters left verbatim rather than encoded. "=" is absent from the sets
+# above, so that "%3D" is decoded into it and both spellings compare equal.
+_SAFE = f"{_KEPT}="
+_PATTERN_SAFE = f"{_PATTERN_KEPT}="
+
+# A URL whose path is made only of characters that percent-encoding leaves
+# alone, and that urlsplit does not read as a delimiter, needs no quoting. A
+# path starting with "//" is left out because urlsplit reads it as an
+# authority.
+_QUOTED_URL = re.compile(
+ r"(?:[A-Za-z][A-Za-z0-9+.-]*://[^/?#]*)?(/(?!/)[A-Za-z0-9_.~/=-]*)"
+).fullmatch
+
+# Patterns keep "*" unquoted as well, and their trailing "$" as it is.
+_QUOTED_PATTERN = re.compile(r"[A-Za-z0-9_.~/=*-]+\$?").fullmatch
+
+
+def _parse_time_of_day(value: str) -> time:
+ """Parse an HMM or HHMM time-of-day value."""
+ value = value.strip()
+ if not (3 <= len(value) <= 4 and value.isascii() and value.isdigit()):
+ raise ValueError(f"Invalid time of day: {value!r}")
+ return time(int(value[:-2]), int(value[-2:]))
+
def _parse_time_period(time_period: str, separator: str = "-") -> tuple[time,
time]:
"""Parse a string with a time period into a tuple of start and end
times."""
start_time_str, end_time_str = time_period.split(separator)
- start_time = time(int(start_time_str[:2]), int(start_time_str[-2:]))
- end_time = time(int(end_time_str[:2]), int(end_time_str[-2:]))
- return start_time, end_time
+ return _parse_time_of_day(start_time_str), _parse_time_of_day(end_time_str)
def _unquote(url: str, ignore: str = "", errors: str = "replace") -> str:
@@ -48,35 +78,53 @@
return f"%{ord(char):02X}"
-def _quote_path(path: str) -> str:
- """Return percent encoded path."""
- parts = urlparse(path)
- path = _unquote(parts.path, ignore="/%")
- path = quote(path, safe="/%")
-
- parts = ParseResult("", "", path, parts.params, parts.query,
parts.fragment)
- path = urlunparse(parts)
- return path or "/"
+def _quote(value: str, kept: str, safe: str) -> str:
+ """Return *value* with its percent-encoding normalized.
+
+ Escapes other than those of the characters in *kept* are decoded,
+ characters that need an escape are encoded, and those in *safe* are left
+ verbatim. Both sides of a comparison must be normalized with the same
+ *kept* and *safe* sets, or equivalent strings end up spelled differently.
+ """
+ return quote(_unquote(value, ignore=kept), safe=safe)
+
+
+def _quote_path(url: str) -> str:
+ """Return the path and query of *url*, normalized for comparison.
+
+ The scheme, the authority and the fragment are dropped, since rules never
+ match against them.
+ """
+ quoted = _QUOTED_URL(url)
+ if quoted:
+ return quoted[1]
+
+ url = url.partition("#")[0]
+ parts = urlsplit(url)
+ path = parts.path
+ # A "?" with nothing after it is still part of what rules match against.
+ if "?" in url:
+ path += f"?{parts.query}"
+ path = _quote(path, _KEPT, _SAFE)
+ return path if path.startswith("/") else f"/{path}"
def _quote_pattern(pattern: str) -> str:
+ """Return *pattern* normalized for comparison.
+
+ A trailing "$", which anchors the pattern to the end of the URL, is kept
+ as it is; a "$" anywhere else is an ordinary character.
+ """
+ if _QUOTED_PATTERN(pattern):
+ return pattern
+
+ # A rule written as an absolute URL matches a path with that URL in it,
+ # which is what the site owner who wrote it by mistake gets.
if pattern.startswith(("https://";, "http://";)):
- pattern = "/" + pattern
- if pattern.startswith("//"):
- pattern = "//" + pattern
-
- # Corner case for query only (e.g. '/abc?') and param only (e.g. '/abc;')
URLs.
- # Save the last character otherwise, urlparse will kill it.
- last_char = ""
- if pattern[-1] == "?" or pattern[-1] == ";" or pattern[-1] == "$":
- last_char = pattern[-1]
- pattern = pattern[:-1]
+ pattern = f"/{pattern}"
- parts = urlparse(pattern)
- pattern = _unquote(parts.path, ignore="/*$%")
- pattern = quote(pattern, safe="/*%=")
-
- parts = ParseResult(
- "", "", pattern + last_char, parts.params, parts.query, parts.fragment
- )
- return urlunparse(parts)
+ anchor = ""
+ if pattern.endswith("$"):
+ anchor = "$"
+ pattern = pattern[:-1]
+ return _quote(pattern, _PATTERN_KEPT, _PATTERN_SAFE) + anchor
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/protego-0.6.2/tests/test_on_google_spec.py
new/protego-0.7.0/tests/test_on_google_spec.py
--- old/protego-0.6.2/tests/test_on_google_spec.py 2020-02-02
01:00:00.000000000 +0100
+++ new/protego-0.7.0/tests/test_on_google_spec.py 2020-02-02
01:00:00.000000000 +0100
@@ -39,6 +39,32 @@
@pytest.mark.parametrize(
+ ("robots_user_agent", "crawler_user_agent", "applies"),
+ [
+ # A product token must not match as a substring in the middle of
+ # another token (https://github.com/scrapy/protego/issues/65).
+ ("bot", "mybot", False),
+ ("google", "notgooglebot", False),
+ # Exact matches and boundary prefixes do apply.
+ ("googlebot", "googlebot", True),
+ ("googlebot", "googlebot-news", True),
+ ("google", "googlebot-mobile", True),
+ # A token embedded in a full User-Agent header still applies.
+ ("foobot", "Mozilla/5.0 (compatible; Foobot/1.0)", True),
+ # Matching is case-insensitive.
+ ("foobot", "FOOBOT", True),
+ ],
+)
+def test_user_agent_boundary_matching(robots_user_agent, crawler_user_agent,
applies):
+ content = f"User-Agent: {robots_user_agent}\nDisallow: /private\n"
+ rp = Protego.parse(content=content)
+ # When the group applies, /private is disallowed for the crawler.
+ assert (
+ not rp.can_fetch("http://example.com/private";, crawler_user_agent)
+ ) == applies
+
+
[email protected](
("pattern", "path", "match"),
[
("/", "/harry", True),
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/protego-0.6.2/tests/test_protego.py
new/protego-0.7.0/tests/test_protego.py
--- old/protego-0.6.2/tests/test_protego.py 2020-02-02 01:00:00.000000000
+0100
+++ new/protego-0.7.0/tests/test_protego.py 2020-02-02 01:00:00.000000000
+0100
@@ -3,7 +3,7 @@
import pytest
from protego import Protego
-from protego._utils import _parse_time_period
+from protego._utils import _parse_time_period, _quote_path, _quote_pattern
class TestProtego:
@@ -179,28 +179,32 @@
rp = Protego.parse(content=content)
assert rp.preferred_host is None
- def test_crawl_delay(self):
+ @pytest.mark.parametrize("value", [0, 10, 15])
+ def test_crawl_delay(self, value: int) -> None:
content = (
"User-agent: * \n"
"Disallow: /disallowed \n"
"Allow: /allowed \n"
"Crawl-delay: 10 \n"
"User-agent: testbot\n"
- "Crawl-delay: 15 \n"
+ f"Crawl-delay: {value}\n"
)
rp = Protego.parse(content=content)
assert rp.crawl_delay("*") == 10.0
- assert rp.crawl_delay("testbot") == 15.0
+ assert rp.crawl_delay("testbot") == value
- def test_malformed_crawl_delay(self):
+ @pytest.mark.parametrize(
+ "value", ["random_word", "-5", "-0.5", "inf", "-inf", "nan"]
+ )
+ def test_malformed_crawl_delay(self, value: str) -> None:
content = (
"User-agent: * \n"
"Disallow: /disallowed \n"
"Allow: /allowed \n"
- "Crawl-delay: random_word"
+ f"Crawl-delay: {value}"
)
rp = Protego.parse(content=content)
- assert rp.crawl_delay("*") is None
+ assert rp.crawl_delay("*") is None, value
def test_no_crawl_delay(self):
content = "User-agent: * \nDisallow: /disallowed \nAllow: /allowed"
@@ -317,6 +321,25 @@
rp = Protego.parse(content=content)
assert rp.request_rate("two") is None
+ @pytest.mark.parametrize(
+ "value",
+ [
+ "one/two",
+ "5",
+ "-1/5",
+ "0/5",
+ "1/0",
+ "0/0",
+ "1/-5",
+ # A malformed time window invalidates the whole rule.
+ "1/5 9-17",
+ ],
+ )
+ def test_malformed_request_rate(self, value):
+ content = f"User-agent: *\nRequest-rate: {value}"
+ rp = Protego.parse(content=content)
+ assert rp.request_rate("*") is None
+
def test_empty_response(self):
"""empty response should equal 'allow all'"""
rp = Protego.parse(content="")
@@ -325,6 +348,21 @@
assert rp.can_fetch("https://site.local/index.html";, "*")
assert rp.can_fetch("https://site.local/disallowed";, "*")
+ def test_wildcard_rule_in_a_rule_set_with_plain_rules(self):
+ content = """User-agent: *
+ Disallow: /admin/
+ Disallow: /private/
+ Disallow: /*.pdf
+ Disallow: /tmp$
+ """
+ rp = Protego.parse(content=content)
+
+ assert not rp.can_fetch("https://site.local/docs/manual.pdf";, "bot")
+ assert not rp.can_fetch("https://site.local/tmp";, "bot")
+ assert not rp.can_fetch("https://site.local/admin/index.html";, "bot")
+ assert rp.can_fetch("https://site.local/docs/manual.txt";, "bot")
+ assert rp.can_fetch("https://site.local/tmp/file";, "bot")
+
def test_allowed_wildcards(self):
content = """User-agent: first
Disallow: /disallowed/*/end$
@@ -655,6 +693,21 @@
assert rp.can_fetch("https://site.local/path1";, "two")
assert rp.can_fetch("https://site.local/path2";, "two")
+ def test_directives_without_path(self):
+ """Values whose path is empty after quoting, such as a bare URL
+ without a path, add no rule."""
+ content = """
+ User-Agent: one
+ Disallow: ftp://site.local
+
+ User-Agent: two
+ Allow: ftp://site.local
+ Disallow: /
+ """
+ rp = Protego.parse(content=content)
+ assert rp.can_fetch("https://site.local/path";, "one")
+ assert not rp.can_fetch("https://site.local/path";, "two")
+
def test_empty_record_group(self):
content = """
User-Agent: harrybot
@@ -665,6 +718,108 @@
assert rp.can_fetch("https://site.local/path1";, "testbot")
assert rp.can_fetch("https://site.local/path2";, "testbot")
+ def test_empty_user_agent_line(self):
+ """A user agent line with no value still starts a new record group."""
+ content = """
+ User-Agent: alpha
+ Disallow: /x
+ User-Agent:
+ User-Agent: beta
+ Disallow: /y
+ """
+ rp = Protego.parse(content=content)
+ assert not rp.can_fetch("https://site.local/x";, "alpha")
+ assert rp.can_fetch("https://site.local/y";, "alpha")
+ assert rp.can_fetch("https://site.local/x";, "beta")
+ assert not rp.can_fetch("https://site.local/y";, "beta")
+
+ # Rules following a lone empty user agent line apply to no one.
+ content = """
+ User-Agent: alpha
+ Disallow: /x
+ User-Agent:
+ Disallow: /y
+ """
+ rp = Protego.parse(content=content)
+ assert not rp.can_fetch("https://site.local/x";, "alpha")
+ assert rp.can_fetch("https://site.local/y";, "alpha")
+
+ # An empty user agent line between two user agent lines does not
+ # split their group.
+ content = """
+ User-Agent: alpha
+ User-Agent:
+ User-Agent: beta
+ Disallow: /x
+ """
+ rp = Protego.parse(content=content)
+ assert not rp.can_fetch("https://site.local/x";, "alpha")
+ assert not rp.can_fetch("https://site.local/x";, "beta")
+
+ # A colon-less, value-less user agent line behaves the same as
+ # "User-Agent:".
+ content = """
+ User-Agent: alpha
+ Disallow: /x
+ user agent
+ Disallow: /y
+ """
+ rp = Protego.parse(content=content)
+ assert not rp.can_fetch("https://site.local/x";, "alpha")
+ assert rp.can_fetch("https://site.local/y";, "alpha")
+
+ def test_global_directives_do_not_split_groups(self):
+ """Sitemap and Host lines are site-wide; consecutive user agent lines
+ form a single group across them."""
+ content = """
+ User-Agent: one
+ Sitemap: https://site.local/sitemap.xml
+ User-Agent: two
+ Host: www.site.local
+ User-Agent: three
+ Disallow: /disallowed
+ """
+ rp = Protego.parse(content=content)
+ assert not rp.can_fetch("https://site.local/disallowed";, "one")
+ assert not rp.can_fetch("https://site.local/disallowed";, "two")
+ assert not rp.can_fetch("https://site.local/disallowed";, "three")
+ assert list(rp.sitemaps) == ["https://site.local/sitemap.xml";]
+ assert rp.preferred_host == "www.site.local"
+
+ # They do not prevent a legitimate group split either.
+ content = """
+ User-Agent: one
+ Disallow: /disallowed
+ Sitemap: https://site.local/sitemap.xml
+ User-Agent: two
+ Crawl-delay: 5
+ """
+ rp = Protego.parse(content=content)
+ assert not rp.can_fetch("https://site.local/disallowed";, "one")
+ assert rp.can_fetch("https://site.local/disallowed";, "two")
+ assert rp.crawl_delay("one") is None
+ assert rp.crawl_delay("two") == 5.0
+
+ def test_preferred_host_before_user_agent(self):
+ """Host, like Sitemap, is a site-wide directive and needs no
+ preceding user agent line."""
+ content = """
+ Host: www.site.local
+ Sitemap: https://site.local/sitemap.xml
+ """
+ rp = Protego.parse(content=content)
+ assert rp.preferred_host == "www.site.local"
+ assert list(rp.sitemaps) == ["https://site.local/sitemap.xml";]
+
+ content = """
+ Host: www.site.local
+ User-Agent: *
+ Disallow: /disallowed
+ """
+ rp = Protego.parse(content=content)
+ assert rp.preferred_host == "www.site.local"
+ assert not rp.can_fetch("https://site.local/disallowed";, "FooBot")
+
def test_1994rfc_example(self):
"""Test parser on examples form 1994 RFC."""
content = """
@@ -823,6 +978,16 @@
assert not rp.can_fetch("/", "SomeOtherBot")
assert not rp.can_fetch("/blahblahblah", "SomeOtherBot")
+ def test_robotstxt_is_always_allowed(self):
+ content = """
+ User-agent: *
+ Disallow: /
+ """
+ rp = Protego.parse(content=content)
+ assert rp.can_fetch("/robots.txt", "foobot")
+ assert rp.can_fetch("http://www.example.com/robots.txt";, "foobot")
+ assert not rp.can_fetch("http://www.example.com/a/robots.txt";,
"foobot")
+
def test_grouping_unknown_keys(self):
"""
When we encounter unknown keys, we should disregard any grouping that
may have
@@ -886,6 +1051,103 @@
assert not rp.can_fetch("http://foo.bar/myprofile";, "foo*bot")
assert not rp.can_fetch("http://foo.bar/myprofile";, "foobot")
+ def test_generosity_case_insensitivity(self):
+ """Colon-less directive lines are salvaged regardless of case."""
+ content = """
+ User-Agent FooBot
+ Disallow /disallowed
+ Allow /disallowed/exception
+ Crawl-delay 5
+ """
+ rp = Protego.parse(content=content)
+ assert not rp.can_fetch("https://site.local/disallowed";, "FooBot")
+ assert rp.can_fetch("https://site.local/disallowed/exception";,
"FooBot")
+ assert rp.can_fetch("https://site.local/disallowed";, "BarBot")
+ assert rp.crawl_delay("FooBot") == 5.0
+
+ def test_generosity_visit_time(self):
+ """Colon-less Visit-time lines are salvaged like other directives."""
+ content = """
+ User-agent: *
+ visit-time 0200 0630
+ """
+ rp = Protego.parse(content=content)
+ visit_time = rp.visit_time("FooBot")
+ assert visit_time is not None
+ assert visit_time.start_time == time(2, 0)
+ assert visit_time.end_time == time(6, 30)
+
+ def test_generosity_sitemap(self):
+ """A colon-less Sitemap line still contains a colon in its URL value,
+ so it is parsed as "<field>:<value>" with a nonsense field before the
+ space-separated fallback can salvage it."""
+ content = """
+ User-agent: *
+ Disallow: /disallowed
+ Sitemap https://site.local/sitemap.xml
+ """
+ rp = Protego.parse(content=content)
+ assert list(rp.sitemaps) == ["https://site.local/sitemap.xml";]
+ assert not rp.can_fetch("https://site.local/disallowed";, "FooBot")
+
+ def test_generosity_not_for_unknown_colon_directives(self):
+ """A line with an unknown "<field>: <value>" directive is ignored,
+ even if it starts with a directive name; only a colon that cannot be
+ a field separator (e.g. inside a URL) triggers the space-separated
+ fallback."""
+ content = """
+ User-agent: *
+ Disallow: /disallowed
+ User-agent must match exactly: see the docs
+ Allow: /disallowed/exception
+ """
+ rp = Protego.parse(content=content)
+ assert not rp.can_fetch("https://site.local/disallowed";, "FooBot")
+ assert rp.can_fetch("https://site.local/disallowed/exception";,
"FooBot")
+
+ content = """
+ Host: www.site.local
+ Host name: www.other.example
+ Sitemap index: see https://site.local/sitemap.xml
+ """
+ rp = Protego.parse(content=content)
+ assert rp.preferred_host == "www.site.local"
+ assert list(rp.sitemaps) == []
+
+ def test_generosity_tab_separated(self):
+ """Colon-less directive lines may use any whitespace as the
+ field-value separator."""
+ content = "User-agent: *\nDisallow\t/disallowed"
+ rp = Protego.parse(content=content)
+ assert not rp.can_fetch("https://site.local/disallowed";, "FooBot")
+
+ def test_empty_field_line_does_not_merge_groups(self):
+ """A line with an empty field (e.g. a stray ":") is ignored and does
+ not prevent the next user agent line from starting a new group."""
+ content = """
+ User-Agent: alpha
+ Disallow: /x
+ :
+ User-Agent: beta
+ Disallow: /y
+ """
+ rp = Protego.parse(content=content)
+ assert not rp.can_fetch("https://site.local/x";, "alpha")
+ assert rp.can_fetch("https://site.local/y";, "alpha")
+ assert not rp.can_fetch("https://site.local/y";, "beta")
+
+ content = """
+ User-Agent: alpha
+ Disallow: /x
+ : foo
+ User-Agent: beta
+ Disallow: /y
+ """
+ rp = Protego.parse(content=content)
+ assert not rp.can_fetch("https://site.local/x";, "alpha")
+ assert rp.can_fetch("https://site.local/y";, "alpha")
+ assert not rp.can_fetch("https://site.local/y";, "beta")
+
def test_directive_case_insensitivity(self):
content = """
USER-AGENT: onebot
@@ -957,6 +1219,9 @@
assert rp.can_fetch("http://foo.com/allowed-slash/index.html";,
"footbot")
assert not rp.can_fetch("http://foo.com/allowed-slash/index.htm";,
"footbot")
assert not rp.can_fetch("http://foo.com/anyother-url";, "footbot")
+ # The "$" anchor of the added directory rule must not leak a rule
+ # matching a literal dollar sign.
+ assert not rp.can_fetch("http://foo.com/allowed-slash/$budget";,
"footbot")
def test_percentage_encoding(self):
content = (
@@ -984,6 +1249,29 @@
assert rp.can_fetch("http://foo.bar/foo/bar/baz";, "FooBot")
assert rp.can_fetch("http://foo.bar/foo/bar/%62%61%7A";, "FooBot")
+ def test_percentage_encoding_query(self):
+ """Queries are encoded like paths, so that a rule matches a URL
+ regardless of how either of the two spells a character."""
+ content = "User-agent: FooBot\nDisallow: /en-uk/*q=*relevance*\n"
+ rp = Protego.parse(content=content)
+ assert not rp.can_fetch(
+ "http://foo.bar/en-uk/c/4018?q=%3Arelevance%3Atype%3AFood";,
"FooBot"
+ )
+ assert not rp.can_fetch("http://foo.bar/en-uk/c/4018?q=:relevance";,
"FooBot")
+ assert rp.can_fetch("http://foo.bar/en-uk/c/4018?q=:price";, "FooBot")
+
+ content = "User-agent: FooBot\nDisallow: /\nAllow: /foo?bar=ツ\n"
+ rp = Protego.parse(content=content)
+ assert rp.can_fetch("http://foo.bar/foo?bar=ツ";, "FooBot")
+ assert rp.can_fetch("http://foo.bar/foo?bar=%E3%83%84";, "FooBot")
+ assert rp.can_fetch("http://foo.bar/foo?bar=%e3%83%84";, "FooBot")
+
+ # A parameter and a query are matched against, a fragment is not.
+ content = "User-agent: FooBot\nDisallow: /foo;a=b?c=d$\n"
+ rp = Protego.parse(content=content)
+ assert not rp.can_fetch("http://foo.bar/foo;a=b?c=d";, "FooBot")
+ assert not rp.can_fetch("http://foo.bar/foo;a=b?c=d#e";, "FooBot")
+
def test_url_case_sensitivity(self):
content = "user-agent: FooBot\ndisallow: /x/\n"
rp = Protego.parse(content=content)
@@ -1058,7 +1346,7 @@
def test_bytestrings(self):
content = b"User-Agent: FootBot\nDisallow: /something"
- with pytest.raises(ValueError, match=r"Protego\.parse expects str, got
bytes"):
+ with pytest.raises(TypeError, match=r"Protego\.parse expects str, got
bytes"):
Protego.parse(content=content) # type: ignore[arg-type]
def test_leading_double_slash_in_pattern(self):
@@ -1072,14 +1360,39 @@
assert not rp.can_fetch("http://example.com//folder/page";, "FooBot")
def test_visit_time(self):
- """Some website specified allow time for crawling in UTC"""
- content = "User-Agent: *\nVisit-time: 0200 0630\nUser-Agent: NoTime"
+ """The de-facto Visit-time convention, from the "Extended Standard
+ for Robot Exclusion", is hyphen-separated."""
+ content = "User-Agent: *\nVisit-time: 0400-0845\nUser-Agent: NoTime"
rp = Protego.parse(content)
- visit_time = rp.visit_time("FooBoot")
+ visit_time = rp.visit_time("FooBot")
+ assert visit_time is not None
+ assert visit_time.start_time == time(4, 0)
+ assert visit_time.end_time == time(8, 45)
+ assert rp.visit_time("NoTime") is None
+
+ # The space-separated form is also accepted.
+ content = "User-Agent: *\nVisit-time: 0200 0630"
+ rp = Protego.parse(content)
+ visit_time = rp.visit_time("FooBot")
assert visit_time is not None
assert visit_time.start_time == time(2, 0)
assert visit_time.end_time == time(6, 30)
- assert rp.visit_time("NoTime") is None
+
+ @pytest.mark.parametrize(
+ "value",
+ [
+ "9-17",
+ "900",
+ "0400-",
+ "0400-0845-1000",
+ "2500-2600",
+ "abcd-efgh",
+ ],
+ )
+ def test_malformed_visit_time(self, value):
+ content = f"User-Agent: *\nVisit-time: {value}"
+ rp = Protego.parse(content)
+ assert rp.visit_time("FooBot") is None
def test_parse_time_period(self):
start_time, end_time = _parse_time_period("0100-1000")
@@ -1090,6 +1403,15 @@
assert start_time == time(5, 0)
assert end_time == time(6, 0)
+ # Three-digit values are read as HMM.
+ start_time, end_time = _parse_time_period("900-1700")
+ assert start_time == time(9, 0)
+ assert end_time == time(17, 0)
+
+ for time_period in ["9-17", "18-19", "-", "0900-", "090000-100000"]:
+ with pytest.raises(ValueError, match="Invalid time of day"):
+ _parse_time_period(time_period)
+
def test_disallow_query_wildcard(self):
content = "User-agent: * \nDisallow: /*s="
rp = Protego.parse(content=content)
@@ -1099,6 +1421,20 @@
@pytest.mark.parametrize(
+ ("url", "allowed"),
+ [
+ ("https://example.com/1/filter/page=5/";, True),
+ ("https://example.com/1/filter/page%3D5/";, True),
+ ("https://example.com/1/filter/page=5/x";, False),
+ ],
+)
+def test_equal_sign_in_path(url, allowed):
+ content = "User-agent: *\nAllow: /*/filter/page=*/$\nDisallow: /\n"
+ rp = Protego.parse(content)
+ assert rp.can_fetch(url, "*") == allowed
+
+
[email protected](
("allow", "disallow", "url", "allowed"),
[
("*/p", "/", "http://example.com/page";, True),
@@ -1121,3 +1457,29 @@
content = f"User-agent: *\nDisallow: {disallow}\n"
rp = Protego.parse(content)
assert rp.can_fetch(url, "*")
+
+
[email protected](
+ ("value", "path", "pattern"),
+ [
+ ("/a/b", "/a/b", "/a/b"),
+ ("/a=b", "/a=b", "/a=b"),
+ ("//host/path", "/path", "//host/path"),
+ ("/a%2fb", "/a%2Fb", "/a%2Fb"),
+ ("/a?b", "/a%3Fb", "/a%3Fb"),
+ ("/a;b", "/a%3Bb", "/a%3Bb"),
+ ("/a:b", "/a%3Ab", "/a%3Ab"),
+ ("/a b", "/a%20b", "/a%20b"),
+ ("/á", "/%C3%A1", "/%C3%A1"),
+ ("/a*b", "/a%2Ab", "/a*b"),
+ ("/a$b", "/a%24b", "/a%24b"),
+ ("/a$", "/a%24", "/a$"),
+ ("https://example.com/a/b";, "/a/b", "/https%3A//example.com/a/b"),
+ ("https://example.com/a b", "/a%20b", "/https%3A//example.com/a%20b"),
+ ("https://example.com//a";, "//a", "/https%3A//example.com//a"),
+ ("https://example.com";, "/", "/https%3A//example.com"),
+ ],
+)
+def test_quoting(value, path, pattern):
+ assert _quote_path(value) == path
+ assert _quote_pattern(value) == pattern
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/protego-0.6.2/tox.ini new/protego-0.7.0/tox.ini
--- old/protego-0.6.2/tox.ini 2020-02-02 01:00:00.000000000 +0100
+++ new/protego-0.7.0/tox.ini 2020-02-02 01:00:00.000000000 +0100
@@ -7,7 +7,34 @@
pytest-cov >= 7.0.0
pytest-timeout
commands =
- pytest --cov=protego --cov-report=xml --cov-report= {posargs:tests}
+ pytest --cov=protego --cov-report=xml --cov-report=
--cov-report=term-missing \
+ --junitxml=testenv.junit.xml -o junit_family=legacy \
+ {posargs:tests}
+
+[testenv:benchmarks]
+deps =
+ pytest
+ pytest-codspeed==5.0.3
+# The CodSpeed runner drives pytest-codspeed through CODSPEED_* variables,
which
+# have to reach the command tox runs for any measurement to happen.
+passenv =
+ *codspeed*
+ *ci*
+# Hash randomization perturbs dict and set behaviour, which shows up as
+# jitter in instruction counts.
+set_env =
+ PYTHONHASHSEED = 0
+commands =
+ pytest --codspeed {posargs:benchmarks/test_benchmarks.py}
+
+[testenv:comparison]
+deps =
+ -r benchmarks/requirements.txt
+ docutils
+ pytest
+commands =
+ pytest benchmarks/test_comparison.py
+ python -m benchmarks.compare {posargs}
[testenv:pre-commit]
deps = pre-commit
@@ -17,17 +44,17 @@
[testenv:twinecheck]
basepython = python3
deps =
- twine==6.2.0
- build==1.4.0
+ twine==7.0.0
+ build==1.5.0
commands =
python -m build --sdist
twine check dist/*
-[testenv:typing]
+[testenv:mypy]
basepython = python3
deps =
- mypy==1.19.1
- pytest
- scrapy==2.14.1
+ mypy==2.3.1
+ pytest==9.1.1
+ scrapy==2.18.0
commands =
mypy {posargs: src tests}
