[issue19216] stat cache for import bootstrap

STINNER Victor added the comment: The benefit of avoiding stat() calls seems to not be obvious to everybody. Moreover, importlib now implements a "path cache". I close the issue. The most efficient solution is to pack all your modules and the Python stdlib into a ZIP file: everything is done

[issue19216] stat cache for import bootstrap

Changes by Antoine Pitrou : -- versions: +Python 3.5 -Python 3.4 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue19216] stat cache for import bootstrap

Antoine Pitrou added the comment: > > Also, have you read what I've just posted? > > About the fuzziness of when startup is finished? As implied above, > I'd say at the end of Py_Initialize(). You only have imported a handful of modules by then. Real-world applications will import many more af

[issue19216] stat cache for import bootstrap

Eric Snow added the comment: > I don't really understand the algorithm you're proposing. In importlib._bootstrap: We have some global like "_CHECK_STAT=True". FileFinder would use it to decide on using stat checks or not. In Python/pythonrun.c: At the end of import_init(), we set importlib.

[issue19216] stat cache for import bootstrap

Antoine Pitrou added the comment: > Would it be feasible to have an explicit (but private?) flag in > importlib indicating stat checking (or even all FS checking) should be > disabled, defaulting to True? runpy could set it to False after > initializing importlib and then back to True when start

[issue19216] stat cache for import bootstrap

Eric Snow added the comment: Would it be feasible to have an explicit (but private?) flag in importlib indicating stat checking (or even all FS checking) should be disabled, defaulting to True? runpy could set it to False after initializing importlib and then back to True when startup is done

[issue19216] stat cache for import bootstrap

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue19216] stat cache for import bootstrap

Antoine Pitrou added the comment: The real problem here is that the definition of "bootstrap" or "startup" is fuzzy. How do you decide when you stop caching? The only workable approach IMO is to adopt a time-based heuristic, which I did in issue14067. -- __

[issue19216] stat cache for import bootstrap

Changes by Brett Cannon : -- assignee: brett.cannon -> ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:/

[issue19216] stat cache for import bootstrap

Eric Snow added the comment: I forgot to mention that optimizing the default composition of sys.path (from site) could help speed things up, though it might already be optimized in that regard. I also forgot to mention the idea of zipping up the stdlib. Sorry for the sidetrack. Now, back to

[issue19216] stat cache for import bootstrap

Eric Snow added the comment: I realized those two stats are not superfluous in the case that a directory name has a .py suffix or a file doesn't have any suffix. However, I expect that's pretty uncommon. Worst case, these cases cost 2 stats per path entry. In practice they cost nothing due

[issue19216] stat cache for import bootstrap

Brett Cannon added the comment: So the 2 stat calls in the general case are superfluous, it's just a question of whether they make any performance difference. Turns out that at least on my Macbook their is no performance difference and thus not worth the cost of breaking semantics over it: htt

[issue19216] stat cache for import bootstrap

Eric Snow added the comment: For interpreter startup, stats are not involved for builtin and frozen modules[1]. They are tied to imports that involve traversing sys.path (a.k.a. PathFinder). Most stats happen in FileFinder.find_loader. The remainder are for source (.py) files (a.k.a. Source

[issue19216] stat cache for import bootstrap

Eric Snow added the comment: With ModuleSpec (PEP 451), the finder creates the spec object (where it stores the loader). At that point the finder is free to store any stat object you like in spec.loader_state. The spec is made available to the loader during exec (if the loader supports it, w

[issue19216] stat cache for import bootstrap

Brett Cannon added the comment: importlib/_bootstrap.py is importlib, period, so there is no separation of what is used to start Python and what is used after interpreter startup is completed. As for adding a 'stat' argument to the loaders, it's possible but as always it comes down to whether

[issue19216] stat cache for import bootstrap

Christian Heimes added the comment: Is the content of the bootstrap module used after the interpreter is boot strapped? I see ... that's a problem. It's a proof of concept anyway and the speed up is minimal. On my computer with a SSD the speedup barely measurable. I'd like to see if it makes a

[issue19216] stat cache for import bootstrap

Brett Cannon added the comment: A cursory look at the patch suggests that the cache use is permanent and so any dynamic changes to a file or directory after an initial caching will not be picked up. Did you run the test suite with this patch as it should have failed. -- __

[issue19216] stat cache for import bootstrap

Antoine Pitrou added the comment: Benchmarks? -- nosy: +pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue19216] stat cache for import bootstrap

STINNER Victor added the comment: See also #14604. -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscrib

[issue19216] stat cache for import bootstrap

Changes by Barry A. Warsaw : -- nosy: +barry ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue19216] stat cache for import bootstrap

Changes by R. David Murray : -- nosy: +r.david.murray ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://m

[issue19216] stat cache for import bootstrap

New submission from Christian Heimes: The import library uses excessive stat() calls. I've implemented a simple cache for the bootstrap module that reduces the amount of stat() calls by almost 1/3 (236 -> 159 on Linux). -- assignee: brett.cannon files: import_stat_cache.patch keywords: