[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library
Jimmy Lai added the comment: Just found Guido mentioned LibCST. Here is a quick overview: 1. LibCST is an open source Python concrete syntax tree parser. It provides a CST looks like and feel like AST. 2. It's built by Instagram for linter and refactoring tools (exact use cases what Łukasz mentioned). We have a linter framework built on top of LibCST which allows a lint rule automatically fixes the issue (autofixer) for developers. We're working on open source it to help developers write better code easily. CC Tim We also found a couple other linter related open source tools use LibCST. 3. It's based on parso (which based on pgen2) and currently supports Python 3.5 to 3.8. Tim is working on adding the support back to 3.0 now and potentially 2.7 later. 4. It provides various patterns for traversing and modifying CST easily, including the AST visitor/transformer pattern, matchers pattern, various helpers for find/replace nodes in a tree and high level transform helpers (e.g. added needed import, remove unused import). 5. It also provides metadata for tree node from static analysis, e.g. line/column position, qualified name, scope analysis, inferred type annotation (through Pyre). Those are useful information for building advanced linter or refactoring tool. There are more features available in LibCST. We continue to develop it to make automated refactoring even easier. We welcome your feedback and PRs! -- nosy: +jimmylai ___ Python tracker <https://bugs.python.org/issue7> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30423] [asyncio] orphan future close loop and cause "RuntimeError: Event loop stopped before Future completed."
Changes by Jimmy Lai : -- pull_requests: +3329 ___ Python tracker <http://bugs.python.org/issue30423> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31350] Optimize get_event_loop and _get_running_loop
New submission from Jimmy Lai: get_event_loop() and _get_running_loop() can be faster. CaseTimeMeanImprove No Change 7.323 +- 0.172 7.323 0.00% Remove class _RunningLoop 6.513 +- 0.115 6.513 -11.06% Expand _get_running_loop() inside get_event_loop() 5.851 +- 0.160 5.851 -20.10% Use Tuple instead of two attributes 6.179 +- 0.099 6.179 -15.62% Tuple + Remove _RunningLoop 6.026 +- 0.123 6.026 -17.71% Tuple + return ternary + Remove _RunningLoop6.060 +- 0.111 6.06-17.25% Combine all four optimizations 4.735 +- 0.111 4.735 -35.34% Remove class _RunningLoop + Use Tuple instead of two attributes 6.241 +- 0.097 6.241 -14.78% Experimenting with different techniques to optimize get_event_loop and _get_running_loop. After discuss with Yuri, decide not to expand _get_running_loop inside get_event_loop. Combine tuple in _running_loop and Remove _RunningLoop (just use threading.local) can achieve the best improvement: 17.71% faster. -- components: asyncio messages: 301342 nosy: jimmylai, yselivanov priority: normal pull_requests: 3360 severity: normal status: open title: Optimize get_event_loop and _get_running_loop versions: Python 3.6, Python 3.7 ___ Python tracker <http://bugs.python.org/issue31350> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31350] Optimize get_event_loop and _get_running_loop
Changes by Jimmy Lai : -- type: -> performance ___ Python tracker <http://bugs.python.org/issue31350> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31350] asyncio: Optimize get_event_loop and _get_running_loop
Jimmy Lai added the comment:
Benchmark script: Run 10 times to get mean and stdev
import asyncio
import time
async def async_get_loop():
start_time = time.time()
for _ in range(500):
asyncio.get_event_loop()
return time.time() - start_time
loop = asyncio.get_event_loop()
results = []
for _ in range(10):
start_time = time.time()
result = loop.run_until_complete(async_get_loop())
results.append(result)
import statistics
print("elapse time: %.3lf +- %.3lf secs" % (statistics.mean(results),
statistics.stdev(results)))
--
___
Python tracker
<http://bugs.python.org/issue31350>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26171] heap overflow in zipimporter module
Changes by Jimmy Lai : -- pull_requests: +1794 ___ Python tracker <http://bugs.python.org/issue26171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30423] [asyncio] orphan future close loop and cause "RuntimeError: Event loop stopped before Future completed."
New submission from Jimmy Lai: Problem: "RuntimeError: Event loop stopped before Future completed." throws when calling run_until_complete(). We investigate and find out some orphan futures stay in the event loop before we run another run_until_complete(another_async_func()). The orphan future has pending state and is attached with _run_until_complete_cb from previous run_until_complete. It happens because the orphan future thrown Exception and then raise, thus remove_done_callback(_run_until_complete_cb) didn't called. Move it to finally section can fix it. With this patch, we stop seeing the Runtime Error. -- components: asyncio messages: 294102 nosy: jimmylai, yselivanov priority: normal pull_requests: 1795 severity: normal status: open title: [asyncio] orphan future close loop and cause "RuntimeError: Event loop stopped before Future completed." versions: Python 3.6 ___ Python tracker <http://bugs.python.org/issue30423> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33505] Optimize asyncio.ensure_future by reordering if conditions
New submission from Jimmy Lai : `ensure_future` converts the input as future if it's not already a future. The condition is the following: if futures.isfuture(coro_or_future): ... elif coroutines.iscoroutine(coro_or_future): ... elif inspect.isawaitable(coro_or_future): ... In real world, `ensure_future` is mostly called by `run_until_complete` and gather with async function call (coroutine) as input. We should check `iscoroutine` first to make it most efficient. -- components: asyncio messages: 316572 nosy: asvetlov, jimmylai, yselivanov priority: normal severity: normal status: open title: Optimize asyncio.ensure_future by reordering if conditions type: performance versions: Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue33505> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33505] Optimize asyncio.ensure_future by reordering if conditions
Change by Jimmy Lai : -- keywords: +patch pull_requests: +6515 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue33505> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33505] Optimize asyncio.ensure_future by reordering if conditions
Change by Jimmy Lai : -- pull_requests: +6518 ___ Python tracker <https://bugs.python.org/issue33505> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33505] Optimize asyncio.ensure_future by reordering if conditions
Jimmy Lai added the comment: Benchmark result: ./python.exe -m perf compare_to ensure_future_original.json ensure_future_optimized.json Mean +- std dev: [ensure_future_original] 57.4 ms +- 4.0 ms -> [ensure_future_optimized] 49.3 ms +- 4.5 ms: 1.17x faster (-14%) -- Added file: https://bugs.python.org/file47591/ensure_future_benchmark.py ___ Python tracker <https://bugs.python.org/issue33505> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33521] Optimize asyncio.isfuture by providing C implementation
New submission from Jimmy Lai : asyncio.isfuture called whenever ensure_future is called. Providing C implementation to make it fast. -- components: asyncio messages: 316670 nosy: asvetlov, jimmylai, yselivanov priority: normal severity: normal status: open title: Optimize asyncio.isfuture by providing C implementation type: performance versions: Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue33521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33521] Add 1.32x faster C implementation of asyncio.isfuture().
Jimmy Lai added the comment: $./python.exe isfuture_benchmark.py -o isfuture_optimized.json $ ./python.exe -m perf compare_to isfuture_original.json isfuture_optimized.json Mean +- std dev: [isfuture_original] 7.16 ms +- 0.23 ms -> [isfuture_optimized] 5.41 ms +- 0.25 ms: 1.32x faster (-24%) -- title: Optimize asyncio.isfuture by providing C implementation -> Add 1.32x faster C implementation of asyncio.isfuture(). Added file: https://bugs.python.org/file47593/isfuture_benchmark.py ___ Python tracker <https://bugs.python.org/issue33521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33521] Add 1.32x faster C implementation of asyncio.isfuture().
Change by Jimmy Lai : -- keywords: +patch pull_requests: +6552 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue33521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33521] Add 1.32x faster C implementation of asyncio.isfuture().
Jimmy Lai added the comment: @vstinner Thanks for the new benchmark, it provides more detailed wins: It's 1.64x faster for future object and 1.23x faster for non-future object. $ ./python.exe -m perf compare_to isfuture_original_2.json isfuture_optimized_2.json future: Mean +- std dev: [isfuture_original_2] 224 ns +- 8 ns -> [isfuture_optimized_2] 135 ns +- 2 ns: 1.66x faster (-40%) task: Mean +- std dev: [isfuture_original_2] 224 ns +- 6 ns -> [isfuture_optimized_2] 137 ns +- 3 ns: 1.64x faster (-39%) regular_func: Mean +- std dev: [isfuture_original_2] 443 ns +- 5 ns -> [isfuture_optimized_2] 361 ns +- 5 ns: 1.23x faster (-18%) str: Mean +- std dev: [isfuture_original_2] 449 ns +- 15 ns -> [isfuture_optimized_2] 360 ns +- 12 ns: 1.25x faster (-20%) -- ___ Python tracker <https://bugs.python.org/issue33521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33521] Add 1.32x faster C implementation of asyncio.isfuture().
Jimmy Lai added the comment: @pitrou This change is part of optimization for asyncio.gather(). gather -> ensure_future -> isfuture/iscoroutine/isawaitable We need C implementation for all those function to make gather really efficient for large scale application (e.g. Instagram) Gather is really slow and cost ~2% CPU on our server. The same optimization approach has been apply on other ciritcal asyncio modules, e.g. Future, get_event_loop, etc. -- ___ Python tracker <https://bugs.python.org/issue33521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33521] Add 1.32x faster C implementation of asyncio.isfuture().
Jimmy Lai added the comment: @pitrou We'll measure the wins of gather when we implement it in C. Before that, we need to get all helpers ready in C. -- ___ Python tracker <https://bugs.python.org/issue33521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33521] Add 1.32x faster C implementation of asyncio.isfuture().
Jimmy Lai added the comment: @vstinner In general, we would like to make all asyncio common functions efficient with C implementation because CPython asyncio overhead is very expensive. In our application, overall it costs about 10% global CPU instructions after we used UVLoop (it's much worse when use default event loop). gather() is only one of the high level function bottleneck. To make CPU overhead not a concern for asyncio user, we should make isfuture in C because it's called by many other event loop functions, e.g. in asyncio/tasks.py, asyncio/coroutines.py, asyncio/base_events.py -- ___ Python tracker <https://bugs.python.org/issue33521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
