[Python-Dev] Second milestone of FAT Python
Hi,
I'm writing a new "FAT Python" project to try to implement optimizations in
CPython (inlining, constant folding, move invariants out of loops, etc.)
using a "static" optimizer (not a JIT). For the background, see the thread
on python-ideas:
https://mail.python.org/pipermail/python-ideas/2015-October/036908.html
See also the documentation:
https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst
I implemented the most basic optimization to test my code: replace calls to
builtin functions (with constant arguments) with the result. For example,
len("abc") is replaced with 3. I reached the second milestone: it's now
possible to run the full Python test suite with these optimizations
enabled. It confirms that the optimizations don't break the Python semantic.
Example:
---
>>> def func():
... return len("abc")
...
>>> import dis
>>> dis.dis(func)
2 0 LOAD_GLOBAL 0 (len)
3 LOAD_CONST 1 ('abc')
6 CALL_FUNCTION1 (1 positional, 0 keyword pair)
9 RETURN_VALUE
>>> len(func.get_specialized())
1
>>> specialized=func.get_specialized()[0]
>>> dis.dis(specialized['code'])
2 0 LOAD_CONST 1 (3)
3 RETURN_VALUE
>>> len(specialized['guards'])
2
>>> func()
3
>>> len=lambda obj: "mock"
>>> func()
'mock'
>>> func.get_specialized()
[]
---
The function func() has specialized bytecode which returns directly 3
instead of calling len("abc"). The specialized bytecode has two guards
dictionary keys: builtins.__dict__['len'] and globals()['len']. If one of
these keys is modified, the specialized bytecode is simply removed (when
the function is called) and the original bytecode is executed.
You cannot expect any speedup at this milestone, it's just to validate the
implementation. You can only get speedup if you implement *manually*
optimizations. See for example posixpath.isabs() which inlines manually the
call to the _get_sep() function. More optimizations will be implemented in
the third milestone. I don't know yet if I will be able to implement
constant folding, function inlining and/or moving invariants out of loops.
Download, compile and test FAT Python with:
hg clone http://hg.python.org/sandbox/fatpython
./configure && make && ./python -m test test_astoptimizer test_fat
Currently, only 24 functions are specialized in the standard library.
Calling a builtin function with constant arguments in not common (it was
expected, it's only the first step for my optimizer). But 161 functions are
specialized in tests.
To be honest, I had to modify some tests to make them pass in FAT mode. But
most changes are related to the .pyc filename, or to the exact size in
bytes of dictionary objects.
FAT Python is still experimental. Currently, the main bug is that the AST
optimizer can optimize a call to a function which is not the expected
builtin function. I already started to implement code to understand
namespaces (detect global and local variables), but it's not enough yet to
detect when a builtin is overriden. See TODO.rst for known bugs and
limitations.
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second milestone of FAT Python
How do you check that someone did not e.g. bind something different to "len"?
On Wed, Nov 4, 2015 at 8:50 AM, Victor Stinner wrote:
> Hi,
>
> I'm writing a new "FAT Python" project to try to implement optimizations in
> CPython (inlining, constant folding, move invariants out of loops, etc.)
> using a "static" optimizer (not a JIT). For the background, see the thread
> on python-ideas:
> https://mail.python.org/pipermail/python-ideas/2015-October/036908.html
>
> See also the documentation:
> https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
> https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst
>
> I implemented the most basic optimization to test my code: replace calls to
> builtin functions (with constant arguments) with the result. For example,
> len("abc") is replaced with 3. I reached the second milestone: it's now
> possible to run the full Python test suite with these optimizations enabled.
> It confirms that the optimizations don't break the Python semantic.
>
> Example:
> ---
def func():
> ... return len("abc")
> ...
import dis
dis.dis(func)
> 2 0 LOAD_GLOBAL 0 (len)
> 3 LOAD_CONST 1 ('abc')
> 6 CALL_FUNCTION1 (1 positional, 0 keyword pair)
> 9 RETURN_VALUE
>
len(func.get_specialized())
> 1
specialized=func.get_specialized()[0]
dis.dis(specialized['code'])
> 2 0 LOAD_CONST 1 (3)
> 3 RETURN_VALUE
len(specialized['guards'])
> 2
>
func()
> 3
>
len=lambda obj: "mock"
func()
> 'mock'
func.get_specialized()
> []
> ---
>
> The function func() has specialized bytecode which returns directly 3
> instead of calling len("abc"). The specialized bytecode has two guards
> dictionary keys: builtins.__dict__['len'] and globals()['len']. If one of
> these keys is modified, the specialized bytecode is simply removed (when the
> function is called) and the original bytecode is executed.
>
>
> You cannot expect any speedup at this milestone, it's just to validate the
> implementation. You can only get speedup if you implement *manually*
> optimizations. See for example posixpath.isabs() which inlines manually the
> call to the _get_sep() function. More optimizations will be implemented in
> the third milestone. I don't know yet if I will be able to implement
> constant folding, function inlining and/or moving invariants out of loops.
>
>
> Download, compile and test FAT Python with:
>
> hg clone http://hg.python.org/sandbox/fatpython
> ./configure && make && ./python -m test test_astoptimizer test_fat
>
>
> Currently, only 24 functions are specialized in the standard library.
> Calling a builtin function with constant arguments in not common (it was
> expected, it's only the first step for my optimizer). But 161 functions are
> specialized in tests.
>
>
> To be honest, I had to modify some tests to make them pass in FAT mode. But
> most changes are related to the .pyc filename, or to the exact size in bytes
> of dictionary objects.
>
> FAT Python is still experimental. Currently, the main bug is that the AST
> optimizer can optimize a call to a function which is not the expected
> builtin function. I already started to implement code to understand
> namespaces (detect global and local variables), but it's not enough yet to
> detect when a builtin is overriden. See TODO.rst for known bugs and
> limitations.
>
> Victor
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second milestone of FAT Python
Uh, sorry, misread your full mail, scratch that
On Wed, Nov 4, 2015 at 9:07 AM, Maciej Fijalkowski wrote:
> How do you check that someone did not e.g. bind something different to "len"?
>
> On Wed, Nov 4, 2015 at 8:50 AM, Victor Stinner
> wrote:
>> Hi,
>>
>> I'm writing a new "FAT Python" project to try to implement optimizations in
>> CPython (inlining, constant folding, move invariants out of loops, etc.)
>> using a "static" optimizer (not a JIT). For the background, see the thread
>> on python-ideas:
>> https://mail.python.org/pipermail/python-ideas/2015-October/036908.html
>>
>> See also the documentation:
>> https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
>> https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst
>>
>> I implemented the most basic optimization to test my code: replace calls to
>> builtin functions (with constant arguments) with the result. For example,
>> len("abc") is replaced with 3. I reached the second milestone: it's now
>> possible to run the full Python test suite with these optimizations enabled.
>> It confirms that the optimizations don't break the Python semantic.
>>
>> Example:
>> ---
> def func():
>> ... return len("abc")
>> ...
> import dis
> dis.dis(func)
>> 2 0 LOAD_GLOBAL 0 (len)
>> 3 LOAD_CONST 1 ('abc')
>> 6 CALL_FUNCTION1 (1 positional, 0 keyword pair)
>> 9 RETURN_VALUE
>>
> len(func.get_specialized())
>> 1
> specialized=func.get_specialized()[0]
> dis.dis(specialized['code'])
>> 2 0 LOAD_CONST 1 (3)
>> 3 RETURN_VALUE
> len(specialized['guards'])
>> 2
>>
> func()
>> 3
>>
> len=lambda obj: "mock"
> func()
>> 'mock'
> func.get_specialized()
>> []
>> ---
>>
>> The function func() has specialized bytecode which returns directly 3
>> instead of calling len("abc"). The specialized bytecode has two guards
>> dictionary keys: builtins.__dict__['len'] and globals()['len']. If one of
>> these keys is modified, the specialized bytecode is simply removed (when the
>> function is called) and the original bytecode is executed.
>>
>>
>> You cannot expect any speedup at this milestone, it's just to validate the
>> implementation. You can only get speedup if you implement *manually*
>> optimizations. See for example posixpath.isabs() which inlines manually the
>> call to the _get_sep() function. More optimizations will be implemented in
>> the third milestone. I don't know yet if I will be able to implement
>> constant folding, function inlining and/or moving invariants out of loops.
>>
>>
>> Download, compile and test FAT Python with:
>>
>> hg clone http://hg.python.org/sandbox/fatpython
>> ./configure && make && ./python -m test test_astoptimizer test_fat
>>
>>
>> Currently, only 24 functions are specialized in the standard library.
>> Calling a builtin function with constant arguments in not common (it was
>> expected, it's only the first step for my optimizer). But 161 functions are
>> specialized in tests.
>>
>>
>> To be honest, I had to modify some tests to make them pass in FAT mode. But
>> most changes are related to the .pyc filename, or to the exact size in bytes
>> of dictionary objects.
>>
>> FAT Python is still experimental. Currently, the main bug is that the AST
>> optimizer can optimize a call to a function which is not the expected
>> builtin function. I already started to implement code to understand
>> namespaces (detect global and local variables), but it's not enough yet to
>> detect when a builtin is overriden. See TODO.rst for known bugs and
>> limitations.
>>
>> Victor
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second milestone of FAT Python
On Wed, Nov 04, 2015 at 09:50:33AM +0100, Victor Stinner wrote: > Hi, > > I'm writing a new "FAT Python" project to try to implement optimizations in > CPython (inlining, constant folding, move invariants out of loops, etc.) > using a "static" optimizer (not a JIT). For the background, see the thread > on python-ideas: > https://mail.python.org/pipermail/python-ideas/2015-October/036908.html Nice work Victor, and thank you for your efforts! -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second milestone of FAT Python
Hi Victor,
great to hear. I think everybody here appreciates your efforts.
Do you think there will be any change of merging this back into CPython?
Best,
Sven
On 04.11.2015 09:50, Victor Stinner wrote:
Hi,
I'm writing a new "FAT Python" project to try to implement
optimizations in CPython (inlining, constant folding, move invariants
out of loops, etc.) using a "static" optimizer (not a JIT). For the
background, see the thread on python-ideas:
https://mail.python.org/pipermail/python-ideas/2015-October/036908.html
See also the documentation:
https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst
I implemented the most basic optimization to test my code: replace
calls to builtin functions (with constant arguments) with the result.
For example, len("abc") is replaced with 3. I reached the second
milestone: it's now possible to run the full Python test suite with
these optimizations enabled. It confirms that the optimizations don't
break the Python semantic.
Example:
---
>>> def func():
... return len("abc")
...
>>> import dis
>>> dis.dis(func)
2 0 LOAD_GLOBAL 0 (len)
3 LOAD_CONST 1 ('abc')
6 CALL_FUNCTION1 (1 positional, 0 keyword pair)
9 RETURN_VALUE
>>> len(func.get_specialized())
1
>>> specialized=func.get_specialized()[0]
>>> dis.dis(specialized['code'])
2 0 LOAD_CONST 1 (3)
3 RETURN_VALUE
>>> len(specialized['guards'])
2
>>> func()
3
>>> len=lambda obj: "mock"
>>> func()
'mock'
>>> func.get_specialized()
[]
---
The function func() has specialized bytecode which returns directly 3
instead of calling len("abc"). The specialized bytecode has two guards
dictionary keys: builtins.__dict__['len'] and globals()['len']. If one
of these keys is modified, the specialized bytecode is simply removed
(when the function is called) and the original bytecode is executed.
You cannot expect any speedup at this milestone, it's just to validate
the implementation. You can only get speedup if you implement
*manually* optimizations. See for example posixpath.isabs() which
inlines manually the call to the _get_sep() function. More
optimizations will be implemented in the third milestone. I don't know
yet if I will be able to implement constant folding, function inlining
and/or moving invariants out of loops.
Download, compile and test FAT Python with:
hg clone http://hg.python.org/sandbox/fatpython
./configure && make && ./python -m test test_astoptimizer test_fat
Currently, only 24 functions are specialized in the standard library.
Calling a builtin function with constant arguments in not common (it
was expected, it's only the first step for my optimizer). But 161
functions are specialized in tests.
To be honest, I had to modify some tests to make them pass in FAT
mode. But most changes are related to the .pyc filename, or to the
exact size in bytes of dictionary objects.
FAT Python is still experimental. Currently, the main bug is that the
AST optimizer can optimize a call to a function which is not the
expected builtin function. I already started to implement code to
understand namespaces (detect global and local variables), but it's
not enough yet to detect when a builtin is overriden. See TODO.rst for
known bugs and limitations.
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second milestone of FAT Python
typo: "chance" instead of "change"
On 04.11.2015 21:14, Sven R. Kunze wrote:
Hi Victor,
great to hear. I think everybody here appreciates your efforts.
Do you think there will be any change of merging this back into CPython?
Best,
Sven
On 04.11.2015 09:50, Victor Stinner wrote:
Hi,
I'm writing a new "FAT Python" project to try to implement
optimizations in CPython (inlining, constant folding, move invariants
out of loops, etc.) using a "static" optimizer (not a JIT). For the
background, see the thread on python-ideas:
https://mail.python.org/pipermail/python-ideas/2015-October/036908.html
See also the documentation:
https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst
I implemented the most basic optimization to test my code: replace
calls to builtin functions (with constant arguments) with the result.
For example, len("abc") is replaced with 3. I reached the second
milestone: it's now possible to run the full Python test suite with
these optimizations enabled. It confirms that the optimizations don't
break the Python semantic.
Example:
---
>>> def func():
... return len("abc")
...
>>> import dis
>>> dis.dis(func)
2 0 LOAD_GLOBAL 0 (len)
3 LOAD_CONST 1 ('abc')
6 CALL_FUNCTION1 (1 positional, 0 keyword pair)
9 RETURN_VALUE
>>> len(func.get_specialized())
1
>>> specialized=func.get_specialized()[0]
>>> dis.dis(specialized['code'])
2 0 LOAD_CONST 1 (3)
3 RETURN_VALUE
>>> len(specialized['guards'])
2
>>> func()
3
>>> len=lambda obj: "mock"
>>> func()
'mock'
>>> func.get_specialized()
[]
---
The function func() has specialized bytecode which returns directly 3
instead of calling len("abc"). The specialized bytecode has two
guards dictionary keys: builtins.__dict__['len'] and
globals()['len']. If one of these keys is modified, the specialized
bytecode is simply removed (when the function is called) and the
original bytecode is executed.
You cannot expect any speedup at this milestone, it's just to
validate the implementation. You can only get speedup if you
implement *manually* optimizations. See for example posixpath.isabs()
which inlines manually the call to the _get_sep() function. More
optimizations will be implemented in the third milestone. I don't
know yet if I will be able to implement constant folding, function
inlining and/or moving invariants out of loops.
Download, compile and test FAT Python with:
hg clone http://hg.python.org/sandbox/fatpython
./configure && make && ./python -m test test_astoptimizer test_fat
Currently, only 24 functions are specialized in the standard library.
Calling a builtin function with constant arguments in not common (it
was expected, it's only the first step for my optimizer). But 161
functions are specialized in tests.
To be honest, I had to modify some tests to make them pass in FAT
mode. But most changes are related to the .pyc filename, or to the
exact size in bytes of dictionary objects.
FAT Python is still experimental. Currently, the main bug is that the
AST optimizer can optimize a call to a function which is not the
expected builtin function. I already started to implement code to
understand namespaces (detect global and local variables), but it's
not enough yet to detect when a builtin is overriden. See TODO.rst
for known bugs and limitations.
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second milestone of FAT Python
On 11/4/2015 3:50 AM, Victor Stinner wrote:
Hi,
I'm writing a new "FAT Python" project to try to implement optimizations
in CPython (inlining, constant folding, move invariants out of loops,
etc.) using a "static" optimizer (not a JIT). For the background, see
the thread on python-ideas:
https://mail.python.org/pipermail/python-ideas/2015-October/036908.html
See also the documentation:
https://hg.python.org/sandbox/fatpython/file/tip/FATPYTHON.rst
https://hg.python.org/sandbox/fatpython/file/tip/ASTOPTIMIZER.rst
I implemented the most basic optimization to test my code: replace calls
to builtin functions (with constant arguments) with the result. For
example, len("abc") is replaced with 3. I reached the second milestone:
it's now possible to run the full Python test suite with these
optimizations enabled. It confirms that the optimizations don't break
the Python semantic.
Is the test suite complete enough to say this? (see below)
Example:
---
>>> def func():
... return len("abc")
...
>>> import dis
>>> dis.dis(func)
2 0 LOAD_GLOBAL 0 (len)
3 LOAD_CONST 1 ('abc')
6 CALL_FUNCTION1 (1 positional, 0 keyword pair)
9 RETURN_VALUE
>>> len(func.get_specialized())
1
>>> specialized=func.get_specialized()[0]
>>> dis.dis(specialized['code'])
2 0 LOAD_CONST 1 (3)
3 RETURN_VALUE
>>> len(specialized['guards'])
2
>>> func()
3
>>> len=lambda obj: "mock"
>>> func()
'mock'
In particular, does the test suite have tests like this, to verify that
over-riding builtins works?
>>> func.get_specialized()
[]
---
--
Terry Jan Reedy
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second milestone of FAT Python
2015-11-04 21:14 GMT+01:00 Sven R. Kunze : > Do you think there will be any change of merging this back into CPython? If I understood correctly the first feedback on python-ideas, there were mostly positive, so I believe that it would be possible to merge the FAT mode into CPython. The changes required to implement the FAT mode are restricted to a few files and are small. But I would prefer to keep the whole "FAT mode" optional, to keep exactly the same performances than before for the "standard" mode. I will write a PEP for that, but first I have to ensure that FAT Python works as expected (it's really faster), and that it's generic enough to be used by projects like Numba, Cython, pythran, etc. (but not JIT-based interpreters like PyPy or pyston which already have their own efficient guards.) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rationale behind lazy map/filter
> I'm not suggesting restarting at the top (I've elsewhere suggested that > many such methods would be better as an *iterable* that can be restarted > at the top by calling iter() multiple times, but that's not the same > thing). I'm suggesting raising an exception other than StopIteration, so > that this situation can be detected. If you are writing code that tries > to resume iterating after the iterator has been exhausted, I have to > ask: why? The most obvious case for me would be tailing a file. Loop over the lines in the file, sleep, then do it again. There are many tasks analogous to that scenario -- anything querying a shared resource. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python 3.5.1 release schedule
At the request of the platform experts, 3.5.1 is now scheduled to happen simultaneously with 2.7.11. That means: Saturday November 21, 2015 tag 3.5.1rc1 Sunday November 22, 2015 release 3.5.1rc1 Saturday December 5, 2015 tag 3.5.1 final Sunday December 6, 2015 release 3.5.1 final I'll update the release schedule soon, //arry/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
