Public bug reported:
I originally wrote this as a comment on
https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1822842 , but I
guess it makes sense to open a new separate bug since the other report was
already closed. Pasting my comment contents below.
~$ lsb_release -rd
Description: Ubuntu 18.04.2 LTS
Release: 18.04
~$ apt-cache policy python3-pip
python3-pip:
Installed: 9.0.1-2.3~ubuntu1.18.04.1
Candidate: 9.0.1-2.3~ubuntu1.18.04.1
Version table:
*** 9.0.1-2.3~ubuntu1.19.04.1 500
500 http://se.archive.ubuntu.com/ubuntu bionic-updates/universe
amd64 packages
500 http://se.archive.ubuntu.com/ubuntu bionic-updates/universe
i386 packages
100 /var/lib/dpkg/status
9.0.1-2 500
500 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64
packages
500 http://se.archive.ubuntu.com/ubuntu bionic/universe i386
packages
---
It seems like `9.0.1-2.3~ubuntu1.18.04.1` breaks the `--extra-index-url`
functionality of PIP.
In my understanding, the idea is for `--extra-index-url` to provide a
PyPI-compliant repository that offers a few additional packages, allowing PIP
to fall back onto the instance configured as `--index-url` (defaults to
upstream PyPI). This has been the case with earlier versions, allowing us to
host an internal PyPI repository containing a subset of packages, as well as a
generic local caching PyPI mirror.
In `9.0.1-2.3~ubuntu1.18.04.1`, it seems like the fallback mechanisms do not
work properly, instead failing if _either_ the `--index-url` _or_
`--extra-index-url` instances lack the package.
---
With `~/.config/pip/pip.conf` as:
[global]
index-url =
https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
extra-index-url =
https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple
With `python3-pip` version `9.0.1-2.3~ubuntu1.18.04.1`, I get a session
like:
~$ python3 -m venv env/pip-latest
~$ . env/pip-latest/bin/activate
(pip-latest) ~$ pip --version
pip 9.0.1 from /home/dandersson/env/pip-latest/lib/python3.6/site-packages
(python 3.6)
(pip-latest) ~$ pip install hpt
Collecting hpt
Exception:
Traceback (most recent call last):
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py",
line 215, in main
status = self.run(options, args)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py",
line 353, in run
wb.build(autobuilding=True)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py",
line 749, in build
self.requirement_set.prepare_files(self.finder)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py",
line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py",
line 554, in _prepare_file
require_hashes
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py",
line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 568, in _get_pages
page = self._get_page(location)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 795, in get_page
resp.raise_for_status()
File
"/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py",
line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url:
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple/hpt/
PIP fails with a traceback due to not finding `hpt` on the `--index-url`
instance -- but the point of giving `--extra-index-url` is that `hpt` resides
on that instance.
Trying to install a package that should be present on the `--index-url`
instance:
(pip-latest) ~$ pip install requests
Collecting requests
Exception:
Traceback (most recent call last):
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py",
line 215, in main
status = self.run(options, args)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py",
line 353, in run
wb.build(autobuilding=True)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py",
line 749, in build
self.requirement_set.prepare_files(self.finder)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py",
line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py",
line 554, in _prepare_file
require_hashes
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py",
line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 568, in _get_pages
page = self._get_page(location)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 795, in get_page
resp.raise_for_status()
File
"/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py",
line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url:
https://artifactory.internal/artifactory/api/pypi/PyPI_extra/simple/requests/
Now it fails with a traceback since `requests` does not exist on the
`--extra-index-url` instance, but the purpose of that instance is to provide a
small subset of extra packages, not to be a full mirror.
Changing `~/.config/pip/pip.conf` to:
[global]
index-url =
https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
removing the `--extra-index-url` instance altogether, it behaves as expected,
not finding the `hpt` package, but being able to install `requests`:
(pip-latest) ~$ pip install hpt
Collecting hpt
Exception:
Traceback (most recent call last):
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/basecommand.py",
line 215, in main
status = self.run(options, args)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/commands/install.py",
line 353, in run
wb.build(autobuilding=True)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/wheel.py",
line 749, in build
self.requirement_set.prepare_files(self.finder)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py",
line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_set.py",
line 554, in _prepare_file
require_hashes
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/req/req_install.py",
line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 568, in _get_pages
page = self._get_page(location)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File
"/home/dandersson/env/pip-latest/lib/python3.6/site-packages/pip/index.py",
line 795, in get_page
resp.raise_for_status()
File
"/home/dandersson/env/pip-latest/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py",
line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url:
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple/hpt/
It is expected to not be able to install `hpt`, but PIP should not fail with a
traceback, but just with a diagnostic message like `No matching distribution
found for hpt`.
Installing a common package will correctly use the `--index-url`
instance:
(pip-latest) ~$ pip install requests
Collecting requests
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: idna, certifi, urllib3, chardet, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8
requests-2.22.0 urllib3-1.25.3
---
Downgrading `python3-pip` to `9.0.1-2`:
~$ sudo apt install python3-pip=9.0.1-2 python-pip-whl=9.0.1-2
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be DOWNGRADED:
python-pip-whl python3-pip
0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 5 not upgraded.
Need to get 1 493 kB of archives.
After this operation, 226 kB disk space will be freed.
Do you want to continue? [Y/n]
Get:1 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pip
all 9.0.1-2 [114 kB]
Get:2 http://se.archive.ubuntu.com/ubuntu bionic/universe amd64
python-pip-whl all 9.0.1-2 [1 379 kB]
Fetched 1 493 kB in 0s (4 177 kB/s)
dpkg: warning: downgrading python3-pip from 9.0.1-2.3~ubuntu1.18.04.1 to
9.0.1-2
(Reading database ... 196581 files and directories currently installed.)
Preparing to unpack .../python3-pip_9.0.1-2_all.deb ...
Unpacking python3-pip (9.0.1-2) over (9.0.1-2.3~ubuntu1.18.04.1) ...
dpkg: warning: downgrading python-pip-whl from 9.0.1-2.3~ubuntu1.18.04.1 to
9.0.1-2
Preparing to unpack .../python-pip-whl_9.0.1-2_all.deb ...
Unpacking python-pip-whl (9.0.1-2) over (9.0.1-2.3~ubuntu1.18.04.1) ...
Setting up python-pip-whl (9.0.1-2) ...
Setting up python3-pip (9.0.1-2) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
and retrying the first attempt with `~/.config/pip/pip.conf` as:
[global]
index-url =
https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
extra-index-url =
https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_extra/simple
yields a session like:
~$ python3 -m venv env/pip-previous
~$ . env/pip-previous/bin/activate
(pip-previous) ~$ pip --version
pip 9.0.1 from
/home/dandersson/env/pip-previous/lib/python3.6/site-packages (python 3.6)
(pip-previous) ~$ pip install hpt
Collecting hpt
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_extra/packages/hpt/0.1.397/hpt-0.1.397.tar.gz
Collecting argh (from hpt)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/06/1c/e667a7126f0b84aaa1c56844337bf0ac12445d1beb9c8a6199a7314944bf/argh-0.26.2-py2.py3-none-any.whl
Collecting pyyaml==5.1 (from hpt)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/9f/2c/9417b5c774792634834e730932745bc09a7d36754ca00acf1ccd1ac2594d/PyYAML-5.1.tar.gz
Collecting argcomplete (from hpt)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/4d/82/f44c9661e479207348a979b1f6f063625d11dc4ca6256af053719bbb0124/argcomplete-1.10.0-py2.py3-none-any.whl
Collecting distro (from hpt)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/ea/35/82f79b92fa4d937146c660a6482cee4f3dfa1f97ff3d2a6f3ecba33e712e/distro-1.4.0-py2.py3-none-any.whl
Collecting requests (from hpt)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting termcolor (from hpt)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
Collecting cilib==v0.1.397 (from hpt)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_extra/packages/cilib/0.1.397/cilib-0.1.397.tar.gz
Collecting idna<2.9,>=2.5 (from requests->hpt)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->hpt)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->hpt)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->hpt)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting hvac (from cilib==v0.1.397->hpt)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/7b/2c/b95b551769459406ca440788c32d699820f4c875020ff95c6e3844eee6d3/hvac-0.9.2-py2.py3-none-any.whl
Collecting psycopg2-binary (from cilib==v0.1.397->hpt)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/b1/35/75c9c2d9cfc073ab6c42b2d8e91ff58c9b99f4ed7ed56b36647642e6080e/psycopg2_binary-2.8.3-cp36-cp36m-manylinux1_x86_64.whl
[...SNIP...]
Successfully installed argcomplete-1.10.0 argh-0.26.2 certifi-2019.6.16
chardet-3.0.4 cilib-0.1.397 distro-1.4.0 hpt-0.1.397 hvac-0.9.2 idna-2.8
psycopg2-binary-2.8.3 pyyaml-5.1 requests-2.22.0 termcolor-1.1.0 urllib3-1.25.3
PIP now correctly resolves the respective packages to their respective
instances, using `--extra-index-url` for the packages that are _only_ present
there, and falling back to `--index-url` for everything else.
Changing `~/.config/pip/pip.conf` to:
[global]
index-url =
https://<user>:<api-token>@artifactory.internal/artifactory/api/pypi/PyPI_mirror/simple
cleaning out the virtual environment and trying again:
(pip-previous) ~$ deactivate
~$ rm -rf env/pip-previous/
~$ python3 -m venv env/pip-previous
~$ . env/pip-previous/bin/activate
(pip-previous) ~$ pip --version
pip 9.0.1 from
/home/dandersson/env/pip-previous/lib/python3.6/site-packages (python 3.6)
(pip-previous) ~$ pip install hpt
Collecting hpt
Could not find a version that satisfies the requirement hpt (from
versions: )
No matching distribution found for hpt
(pip-previous) ~$ pip install requests
Collecting requests
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/69/1b/b853c7a9d4f6a6d00749e94eb6f3a041e342a885b87340b79c1ef73e3a78/certifi-2019.6.16-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
Using cached
https://artifactory.internal/artifactory/api/pypi/PyPI_mirror/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Installing collected packages: chardet, urllib3, certifi, idna, requests
Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8
requests-2.22.0 urllib3-1.25.3
---
In summary, from my point-of-view, the `9.0.1-2.3~ubuntu1.18.04.1` version
breaks all functionality related to `--extra-index-url`, as well as regresses
the behaviour when a package is not found when using `--index-url`.
I will be happy to provide any additional information that can help.
** Affects: python-pip (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1833229
Title:
`--extra-index-url` not working for PIP
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1833229/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
