Hi, everyone. TL;DR: how to deal with setuptools (and newer distutils vendored by setuptools) replacing .egg-info files with directories?
I know I'm reiterating the same topic but I think we're coming to having to actually do something about this and I still haven't found a satisfactory solution. For the people new to the problem, a quick recap: Python packages install metadata called .egg-info (newer build systems use .dist-info but that's another matter). Original Python distutils installed this as a single file but setuptools replaced it with a directory. Now that distutils is deprecated, setuptools is vendoring its own version. We can switch between the system and the vendored version using an envvar. By default, setuptools < 60 uses system distutils, and >= 60 (masked) uses vendored distutils. The big problem is that switching implies changing the format, so if you install foo-X, then switch, then reinstall the same version of foo, you're going to have the file replaced by a directory. This is not supported by the PMS, and Portage handles it somewhat suboptimally (renaming the old file and leaving it orphaned). I should probably emphasize here that the .egg-info path contains the package version, so this is a problem only if the same upstream version is being reinstalled. You can easily reproduce the problem by playing with: SETUPTOOLS_USE_DISTUTILS=stdlib SETUPTOOLS_USE_DISTUTILS=local # vendored and repeatedly building some DISTUTILS_USE_SETUPTOOLS=no packages. >From a quick grep, there are 179 packages using DUS=no right now (and there might be more using distutils without the declaration). I don't think we can ignore the problem. What we can do right now is force the SETUPTOOLS_USE_DISTUTILS=stdlib default back via the eclass or patches for the time being. This will let us finally unmask setuptools-60+ while delaying fixing the problem. However, given pypa's tendency to remove deprecated stuff quickly, this is unlikely to work for long. Some ideas on fixing this: 1. For a start, with distutils deprecated are actually migrating to setuptools or other build systems upstream. This effectively solves the problem for us since the .egg-info switch happens on version bump and there is no file collision. However, this isn't going to help for dead projects. 2. We could control the distutils version in ebuilds directly, i.e. force "stdlib" for the current versions and have developers switch to "local" on version bumps. Combined with 1., this will probably increase the coverage a bit but dead packages will remain in the way. It also relies on all devs understanding the problem. 3. The developers could explicitly bump versions (i.e. create "Gentoo subversions") of packages that don't expect any updates. We can get 100% coverage this way but it's hard and requires patching. 4. We could have the eclasses switch to "local" model and rename the .egg-info files somehow at some point. The main question is "rename how?" 5. We could have the eclasses convert .egg-info into the newer .dist- info format. However, I'm not aware of any existing tool doing such a conversion, and I'm not convinced I want to write one right now, and whether it wouldn't have compatibility implications. These are all the options I can think of right now that don't make my head explode. I'd like to hear your ideas. -- Best regards, Michał Górny
