Re: [Cython] About IndexNode and unicode[index]
>>> >>> I think you could even pass in two flags, one for wraparound and one for >>> boundscheck, and then just evaluate them appropriately in the existing "if" >>> tests above. That should allow both features to be supported independently >>> in a fast way. >>> >> Intresting, could C compilers in optimization mode to eliminate unused >> evaluation path in nested if statements with constant conditional >> expressions? > > They'd be worthless if they didn't do that. (Even Cython does it, BTW.) > Then it can simplify writing utility code in order to support different optimization flags in other cases too. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] About IndexNode and unicode[index]
turn PyUnicode_FromUnicode(PyUnicode_AS_UNICODE(text)+start, stop-start); #endif } unicode_slice.pyx cdef extern from 'unicode_slice.h': inline unicode unicode_slice(unicode ustring, int start, int stop) inline unicode unicode_slice2(unicode ustring, int start, int stop, int flag) cdef unicode text = u"abcdefghigklmnopqrstuvwxyzabcdefghigklmnopqrstuvwxyz" cdef long f_1(unicode text): cdef int i, j cdef int n = len(text) cdef int val cdef long S = 0 for j in range(10): for i in range(n): val = len(unicode_slice(text, 0, i)) S += val * j return S cdef long f_2(unicode text): cdef int i, j cdef int n = len(text) cdef int val cdef long S = 0 for j in range(10): for i in range(n): val = len(unicode_slice2(text, 0, i, 0)) S += val * j return S def test_1(): f_1(text) def test_2(): f_2(text) Here are timings: (py33) zbook:mytests $ python3.3 -m timeit -n 50 -r 5 -s "from mytests.unicode_slice import test_1" "test_1()" 50 loops, best of 5: 534 msec per loop (py33) zbook:mytests $ python3.3 -m timeit -n 50 -r 5 -s "from mytests.unicode_slice import test_2" "test_2()" 50 loops, best of 5: 523 msec per loop Only 2% Zaur Shibzukhov ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] About IndexNode and unicode[index]
>> Then it can simplify writing utility code in order to support >> different optimization flags in other cases too. > > Usually, yes. Look at the dict iteration code, for example, which makes > pretty heavy use of it. > > This may not work in all cases, because the C compiler can decide to *not* > inline a function, for example, or may not be capable of cutting down the > code sufficiently in some specific cases. > > I agree in general, but I wouldn't say that it's worth changing existing > (and working) code. > Thus the strategy of specialization of code to handle special cases, while maintaining the existing code, which works well in general, is the preferred? ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] About IndexNode and unicode[index]
2013/3/1 Stefan Behnel :
> ZS, 28.02.2013 21:07:
>> 2013/2/28 Stefan Behnel:
>>>> This allows to write unicode text parsing code almost at C speed
>>>> mostly in python (+ .pxd defintions).
>>>
>>> I suggest simply adding a constant flag argument to the existing function
>>> that states if checking should be done or not. Inlining will let the C
>>> compiler drop the corresponding code, which may or may nor make it a little
>>> faster.
>>
>> static inline Py_UCS4 unicode_char2(PyObject* ustring, Py_ssize_t i, int
>> flag) {
>> Py_ssize_t length;
>> #if CYTHON_PEP393_ENABLED
>> if (PyUnicode_READY(ustring) < 0) return (Py_UCS4)-1;
>> #endif
>> if (flag) {
>> length = __Pyx_PyUnicode_GET_LENGTH(ustring);
>> if ((0 <= i) & (i < length)) {
>> return __Pyx_PyUnicode_READ_CHAR(ustring, i);
>> } else if ((-length <= i) & (i < 0)) {
>> return __Pyx_PyUnicode_READ_CHAR(ustring, i + length);
>> } else {
>> PyErr_SetString(PyExc_IndexError, "string index out of range");
>> return (Py_UCS4)-1;
>> }
>> } else {
>> return __Pyx_PyUnicode_READ_CHAR(ustring, i);
>> }
>> }
>
> I think you could even pass in two flags, one for wraparound and one for
> boundscheck, and then just evaluate them appropriately in the existing "if"
> tests above. That should allow both features to be supported independently
> in a fast way.
>
>
>> Here are timings:
>>
>> (py33) zbook:mytests $ python3.3 -m timeit -n 50 -r 5 -s "from
>> mytests.unicode_index import test_1" "test_1()"
>> 50 loops, best of 5: 152 msec per loop
>> (py33) zbook:mytests $ python3.3 -m timeit -n 50 -r 5 -s "from
>> mytests.unicode_index import test_2" "test_2()"
>> 50 loops, best of 5: 86.5 msec per loop
>> (py33) zbook:mytests $ python3.3 -m timeit -n 50 -r 5 -s "from
>> mytests.unicode_index import test_3" "test_3()"
>> 50 loops, best of 5: 86.5 msec per loop
>>
>> So your suggestion would be preferable.
>
> Nice. Yes, looks like it' worth it.
>
Could I help in order to include this in 19.0?
Zaur Shibzukhov
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] About IndexNode and unicode[index]
2013/3/1 Stefan Behnel : > Zaur Shibzukhov, 01.03.2013 10:46: >> Could I help in order to include this in 19.0? > > I like pull requests. ;) > OK ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] About IndexNode and unicode[index]
2013/3/2 Stefan Behnel : >> I think you could even pass in two flags, one for wraparound and one for >> boundscheck, and then just evaluate them appropriately in the existing "if" >> tests above. That should allow both features to be supported independently >> in a fast way. > > https://github.com/scoder/cython/commit/cc4f7daec3b1f19b5acaed7766e2b6f86902ad94 It seems to include the following directive at the beginning of the tests (which tests indices for lists, tuples and unicode): #cython: boundscheck=True #cython: wraparound=True as default mode for testing? -- С уважением, Шибзухов З.М. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] About IndexNode and unicode[index]
2013/3/2 Stefan Behnel : > Stefan Behnel, 28.02.2013 22:16: > > https://github.com/scoder/cython/commit/cc4f7daec3b1f19b5acaed7766e2b6f86902ad94 > > Stefan > I tried to build with that change. Tests `unicode_indexing` and `index` are passed. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] To Add datetime.pxd to cython.cpython
2013/3/2 Stefan Behnel : > Hi, > > the last pull request looks good to me now. > > https://github.com/cython/cython/pull/189 > > Any more comments on it? > > Stefan > As was suggested earlier, I added `import_datetime` inline function to initialize PyDateTime C API instead of direct usage of "non-native" C macros from datetime.h. Now you call `import_array ()` first in the same way as is done with `numpy`. This approach looks natural in the light of experience with numpy. Zaur Shibzukhov ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] To Add datetime.pxd to cython.cpython
2013/3/3 Zaur Shibzukhov : > 2013/3/2 Stefan Behnel : >> Hi, >> >> the last pull request looks good to me now. >> >> https://github.com/cython/cython/pull/189 >> >> Any more comments on it? > > As was suggested earlier, I added `import_datetime` inline function to > initialize PyDateTime C API instead of direct usage of "non-native" C > macros from datetime.h. > Now you call `import_array ()` first in the same way as is done with `numpy`. > This approach looks natural in the light of experience with numpy. > I make some performance comparisons. Here example for dates. # test_date.pyx Here test code: from cpython.datetime cimport import_datetime, date_new, date import_datetime() from datetime import date as pydate def test_date1(): cdef list lst = [] for year in range(1000, 2001): for month in range(1,13): for day in range(1, 20): d = pydate(year, month, day) lst.append(d) return lst def test_date2(): cdef list lst = [] for year in range(1000, 2001): for month in range(1,13): for day in range(1, 20): d = date(year, month, day) lst.append(d) return lst def test_date3(): cdef list lst = [] cdef int year, month, day for year in range(1000, 2001): for month in range(1,13): for day in range(1, 20): d = date_new(year, month, day) lst.append(d) return lst def test1(): l = test_date1() return l def test2(): l = test_date2() return l def test3(): l = test_date3() return l Here are timings: (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from mytests.test_date import test1" "test1()" 50 loops, best of 5: 83.2 msec per loop (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from mytests.test_date import test2" "test2()" 50 loops, best of 5: 74.7 msec per loop (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from mytests.test_date import test3" "test3()" 50 loops, best of 5: 20.9 msec per loop OSX 10.6.8 64 bit python 3.2 Shibzukhov Zaur ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] To Add datetime.pxd to cython.cpython
2013/3/3 Zaur Shibzukhov : > 2013/3/3 Zaur Shibzukhov : >> 2013/3/2 Stefan Behnel : >>> Hi, >>> >>> the last pull request looks good to me now. >>> >>> https://github.com/cython/cython/pull/189 >>> >>> Any more comments on it? >> >> As was suggested earlier, I added `import_datetime` inline function to >> initialize PyDateTime C API instead of direct usage of "non-native" C >> macros from datetime.h. >> Now you call `import_array ()` first in the same way as is done with `numpy`. >> This approach looks natural in the light of experience with numpy. >> > I make some performance comparisons. Here example for dates. > > # test_date.pyx > > > Here test code: > > from cpython.datetime cimport import_datetime, date_new, date > > import_datetime() > > from datetime import date as pydate > > def test_date1(): > cdef list lst = [] > for year in range(1000, 2001): > for month in range(1,13): > for day in range(1, 20): > d = pydate(year, month, day) > lst.append(d) > return lst > > > def test_date2(): > cdef list lst = [] > for year in range(1000, 2001): > for month in range(1,13): > for day in range(1, 20): > d = date(year, month, day) > lst.append(d) > return lst > > def test_date3(): > cdef list lst = [] > cdef int year, month, day > for year in range(1000, 2001): > for month in range(1,13): > for day in range(1, 20): > d = date_new(year, month, day) > lst.append(d) > return lst > > def test1(): > l = test_date1() > return l > > def test2(): > l = test_date2() > return l > > def test3(): > l = test_date3() > return l > > Here are timings: > > (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from > mytests.test_date import test1" "test1()" > 50 loops, best of 5: 83.2 msec per loop > (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from > mytests.test_date import test2" "test2()" > 50 loops, best of 5: 74.7 msec per loop > (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from > mytests.test_date import test3" "test3()" > 50 loops, best of 5: 20.9 msec per loop > > OSX 10.6.8 64 bit python 3.2 > More acurate test... # coding: utf-8 from cpython.datetime cimport import_datetime, date_new, date import_datetime() from datetime import date as pydate def test_date1(): cdef list lst = [] cdef int year, month, day for year in range(1000, 2001): for month in range(1,13): for day in range(1, 20): d = pydate(year, month, day) lst.append(d) return lst def test_date2(): cdef list lst = [] cdef int year, month, day for year in range(1000, 2001): for month in range(1,13): for day in range(1, 20): d = date(year, month, day) lst.append(d) return lst def test_date3(): cdef list lst = [] cdef int year, month, day for year in range(1000, 2001): for month in range(1,13): for day in range(1, 20): d = date_new(year, month, day) lst.append(d) return lst def test1(): l = test_date1() return l def test2(): l = test_date2() return l def test3(): l = test_date3() return l Timings: (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from mytests.test_date import test1" "test1()" 50 loops, best of 5: 83.3 msec per loop (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from mytests.test_date import test2" "test2()" 50 loops, best of 5: 74.6 msec per loop (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from mytests.test_date import test3" "test3()" 50 loops, best of 5: 20.8 msec per loop Shibzukhov Zaur ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] To Add datetime.pxd to cython.cpython
2013/3/3 Zaur Shibzukhov : > 2013/3/3 Zaur Shibzukhov : >> 2013/3/3 Zaur Shibzukhov : >>> 2013/3/2 Stefan Behnel : >>>> Hi, >>>> >>>> the last pull request looks good to me now. >>>> >>>> https://github.com/cython/cython/pull/189 >>>> >>>> Any more comments on it? >>> >>> As was suggested earlier, I added `import_datetime` inline function to >>> initialize PyDateTime C API instead of direct usage of "non-native" C >>> macros from datetime.h. >>> Now you call `import_array ()` first in the same way as is done with >>> `numpy`. >>> This approach looks natural in the light of experience with numpy. >>> >> I make some performance comparisons. Here example for dates. >> >> # test_date.pyx >> >> >> Here test code: >> >> from cpython.datetime cimport import_datetime, date_new, date >> >> import_datetime() >> >> from datetime import date as pydate >> >> def test_date1(): >> cdef list lst = [] >> for year in range(1000, 2001): >> for month in range(1,13): >> for day in range(1, 20): >> d = pydate(year, month, day) >> lst.append(d) >> return lst >> >> >> def test_date2(): >> cdef list lst = [] >> for year in range(1000, 2001): >> for month in range(1,13): >> for day in range(1, 20): >> d = date(year, month, day) >> lst.append(d) >> return lst >> >> def test_date3(): >> cdef list lst = [] >> cdef int year, month, day >> for year in range(1000, 2001): >> for month in range(1,13): >> for day in range(1, 20): >> d = date_new(year, month, day) >> lst.append(d) >> return lst >> >> def test1(): >> l = test_date1() >> return l >> >> def test2(): >> l = test_date2() >> return l >> >> def test3(): >> l = test_date3() >> return l >> >> Here are timings: >> >> (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from >> mytests.test_date import test1" "test1()" >> 50 loops, best of 5: 83.2 msec per loop >> (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from >> mytests.test_date import test2" "test2()" >> 50 loops, best of 5: 74.7 msec per loop >> (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from >> mytests.test_date import test3" "test3()" >> 50 loops, best of 5: 20.9 msec per loop >> >> OSX 10.6.8 64 bit python 3.2 >> > > More acurate test... > > # coding: utf-8 > > from cpython.datetime cimport import_datetime, date_new, date > > import_datetime() > > from datetime import date as pydate > > def test_date1(): > cdef list lst = [] > cdef int year, month, day > for year in range(1000, 2001): > for month in range(1,13): > for day in range(1, 20): > d = pydate(year, month, day) > lst.append(d) > return lst > > > def test_date2(): > cdef list lst = [] > cdef int year, month, day > for year in range(1000, 2001): > for month in range(1,13): > for day in range(1, 20): > d = date(year, month, day) > lst.append(d) > return lst > > def test_date3(): > cdef list lst = [] > cdef int year, month, day > for year in range(1000, 2001): > for month in range(1,13): > for day in range(1, 20): > d = date_new(year, month, day) > lst.append(d) > return lst > > def test1(): > l = test_date1() > return l > > def test2(): > l = test_date2() > return l > > def test3(): > l = test_date3() > return l > > Timings: > > (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from > mytests.test_date import test1" "test1()" > 50 loops, best of 5: 83.3 msec per loop > (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from > mytests.test_date import test2" "test2()" > 50 loops, best of 5: 74.6 msec per loop > (py32)zbook:mytests $ python -m timeit -n 50 -r 5 -s "from > mytests.test_date import test3" "test3()" > 50 loops, best of 5: 20.8 msec per loop Yet another performance comp
[Cython] nonecheck and as_none_safe_node method
In ExprNodes.py there are several places where method `as_none_safe_node` was applied in order to wrap a node by NoneCheckNode. I think it would be resonable to apply that mostly only in cases when noncheck=True. Here are possible changes in ExprNodes.py: https://github.com/intellimath/cython/commit/bd041680b78067007ad6b9894a2f2c18514e397c Zaur Shibzukhov ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] nonecheck and as_none_safe_node method
2013/3/5 Zaur Shibzukhov > In ExprNodes.py there are several places where method `as_none_safe_node` > was applied in order to wrap a node by NoneCheckNode. > I think it would be resonable to apply that mostly only in cases when > noncheck=True. > > Here are possible changes in ExprNodes.py: > > https://github.com/intellimath/cython/commit/bd041680b78067007ad6b9894a2f2c18514e397c > > This change would prevent generation of None checking of an objects (lists, tuples, unicode) when nonecheck=True. Any adeas? Zaur Shibzukhov ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] nonecheck and as_none_safe_node method
2013/3/5 Zaur Shibzukhov > 2013/3/5 Zaur Shibzukhov > >> In ExprNodes.py there are several places where method `as_none_safe_node` >> was applied in order to wrap a node by NoneCheckNode. >> I think it would be resonable to apply that mostly only in cases when >> noncheck=True. >> >> Here are possible changes in ExprNodes.py: >> >> https://github.com/intellimath/cython/commit/bd041680b78067007ad6b9894a2f2c18514e397c >> >> This change would prevent generation of None checking of an objects > (lists, tuples, unicode) when nonecheck=True. > Sorry... when nonecheck=False > Any adeas? > > Zaur Shibzukhov -- С уважением, Шибзухов З.М. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] nonecheck and as_none_safe_node method
2013/3/5 Stefan Behnel > Zaur Shibzukhov, 05.03.2013 07:21: > > In ExprNodes.py there are several places where method `as_none_safe_node` > > was applied in order to wrap a node by NoneCheckNode. > > I think it would be resonable to apply that mostly only in cases when > > noncheck=True. > > > > Here are possible changes in ExprNodes.py: > > > https://github.com/intellimath/cython/commit/bd041680b78067007ad6b9894a2f2c18514e397c > > I consider the nonecheck option a quirk. In many, many cases, it's not > obvious to a user to what constructs it applies. For example, we use it to > guard against crashes when we optimise code, e.g. by inlining parts of a > C-API function, when iterating over builtins, etc. In most of these cases, > it depends on more than one parameter if the optimised code will be applied > (and thus no None check) or the fallback, which usually does its own > complete set of safety checks. So it's one of those options that may work > safely in all unit tests and then crash in production. > > Remember, most cases where we leave a None check in the code are not those > where it's obvious that a variable cannot be None because it was just > assigned a non-None value. Most cases are about function arguments, i.e. > values that come from outside of the current function, and thus are not > "obviously" correct even for the human reader or author of the code. > > Also, I'm yet to see a case where a None check really makes a difference in > performance. Often enough, the C compiler will be able to move them out of > loops or drop them completely because it already saw a None check against > the same local variable earlier on. In those cases, it's just Cython not > being smart enough to drop them itself, but without an impact on runtime > performance. And even if the C compiler is not smart enough either, at > least the branch prediction of the processor will strike in the relevant > cases (i.e. inside of loops) and reduce the overhead to "pretty much zero". > > All of this makes em think that we should be very careful when we consider > this option for the generated code. > > I agree that directive nonecheck=False is dangarous in general. This change mainly affect builtin objects (lists, tuples, dicts, unicode) and some situation in function/method calls. And it assume that when you apply this directive you know what you are doing and why. Note that Cython now already set nonecheck=False (but boundcheck and wraparound sets to True in Options.py) by default. But now it not affect builtin types and some special cases. May be the safer strategy is to set nonecheck=True by default and allow locally to apply nonecheck(False) when developer believes that it is necessary? ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] nonecheck and as_none_safe_node method
2013/3/5 Zaur Shibzukhov > > May be the safer strategy is to set nonecheck=True by default and allow > locally to apply nonecheck(False) when developer believes that it is > necessary? > Strategy of making nonecheck=True by default and setting explicitly nonecheck=False when it's necessary is more manageable IMHO. In Cython sources one can explicitly (where it's necessary) ignore this default setting, or set it explicitly to False in concrete context/environment. Zaur Shibzukhov ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython syntax to pre-allocate lists for performance
2013/3/7 Stefan Behnel > Yury V. Zaytsev, 07.03.2013 12:16: > > Is there any syntax that I can use to do something like this in Cython: > > > > py_object_ = PyList_New(123); ? > > Note that Python has an algorithm for shrinking a list on appending, so > this might not be sufficient for your use case. > > > > If not, do you think that this can be added in one way or another? > > > > Unfortunately, I can't think of a non-disruptive way of doing it. For > > instance, if this > > > > [None] * N > > > > is given a completely new meaning, like make an empty list (of NULLs), > > instead of making a real list of Nones, it will certainly break Python > > code. Besides, it would probably be still faster than no pre-allocation, > > but slower than an empty list with pre-allocation... > > > > Maybe > > > > [NULL] * N ? > > What do you need it for? > > Won't list comprehensions work for you? They could potentially be adapted > to presize the list. > > I guess the problem is to construct new (even empty) list with pre-allocated memory exactly for N elements. N*[NULL] - changes semantics because there can't be list with N elements and filled by NULL. N*[None] - more expansive for further assignments because of Py_DECREFs. I suppose that N*[] could do the trick. It could be optimized so that N*[] is equal to an empty list but with preallocated memory exactly for N elements. Could it be? Zaur Shibzukhov ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython syntax to pre-allocate lists for performance
2013/3/7 Zaur Shibzukhov > I guess the problem is to construct new (even empty) list with > pre-allocated memory exactly for N elements. > > N*[NULL] - changes semantics because there can't be list with N elements > and filled by NULL. > N*[None] - more expansive for further assignments because of Py_DECREFs. > > I suppose that N*[] could do the trick. It could be optimized so that N*[] > is equal to an empty list but with preallocated memory exactly for N > elements. Could it be? > > Cython optimize already PyList_Append very well. Theofore scenario when first one create empty list with exactly prealocated memory for N elements and second eval elements of the list and add them using plain list.append could optimized in Cython very well too. As result constructed list will contain memory only for N elements. This allows don't vast memory when one need to build many many lists with relative big size. Zaur Shibzukhov ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Add support for list/tuple slicing
Hello!
Current Cython generate for slicing of list/tuple general
PySequence_GetSlice/SetSlice call.
We could replace that to native call for Py{List|Tuple}_GetSlice and
PyList_SetSlice for lists/tuples.
Here are the changes:
https://github.com/intellimath/cython/commit/27525a5dc9f6eba31b330a6ec04e7a105191d9f5
Zaur Shibzukhov
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Add support for list/tuple slicing
2013/3/7 Zaur Shibzukhov :
> Current Cython generate for slicing of list/tuple general
> PySequence_GetSlice/SetSlice call.
> We could replace that to native call for Py{List|Tuple}_GetSlice and
> PyList_SetSlice for lists/tuples.
There is updated change that use utility code
__Pyx_Py{List|Tuple}_GetSlice because Py{List|Tuple}_GetSlice dosn't
support negative indices. That job do (in CPython) {list|tuple}slice
function from type object's slot ({list|tuple}_subscript), but it
handle both indices and slice objects which add overhead. That's the
reason why PySequence_GetSlice is slower: it create slice object and
falls to {list|tuple}_subscript. Therefore I added utility code.
Here is utility code:
/// PyList_GetSlice.proto ///
static PyObject* __Pyx_PyList_GetSlice(
PyObject* lst, Py_ssize_t start, Py_ssize_t stop);
/// PyList_GetSlice ///
PyObject* __Pyx_PyList_GetSlice(
PyObject* lst, Py_ssize_t start, Py_ssize_t stop) {
Py_ssize_t i, length;
PyListObject* np;
PyObject **src, **dest;
PyObject *v;
length = PyList_GET_SIZE(lst);
if (start < 0) {
start += length;
if (start < 0)
start = 0;
}
if (stop < 0)
stop += length;
else if (stop > length)
stop = length;
length = stop - start;
if (length <= 0)
return PyList_New(0);
np = (PyListObject*) PyList_New(length);
if (np == NULL)
return NULL;
src = ((PyListObject*)lst)->ob_item + start;
dest = np->ob_item;
for (i = 0; i < length; i++) {
v = src[i];
Py_INCREF(v);
dest[i] = v;
}
return (PyObject*)np;
}
/// PyTuple_GetSlice.proto ///
static PyObject* __Pyx_PyTuple_GetSlice(
PyObject* ob, Py_ssize_t start, Py_ssize_t stop);
/// PyTuple_GetSlice ///
PyObject* __Pyx_PyTuple_GetSlice(
PyObject* ob, Py_ssize_t start, Py_ssize_t stop) {
Py_ssize_t i, length;
PyTupleObject* np;
PyObject **src, **dest;
PyObject *v;
length = PyTuple_GET_SIZE(ob);
if (start < 0) {
start += length;
if (start < 0)
start = 0;
}
if (stop < 0)
stop += length;
else if (stop > length)
stop = length;
length = stop - start;
if (length <= 0)
return PyList_New(0);
np = (PyTupleObject *) PyTuple_New(length);
if (np == NULL)
return NULL;
src = ((PyTupleObject*)ob)->ob_item + start;
dest = np->ob_item;
for (i = 0; i < length; i++) {
v = src[i];
Py_INCREF(v);
dest[i] = v;
}
return (PyObject*)np;
}
Here is testing code:
list_slice.pyx
-
from cpython.sequence cimport PySequence_GetSlice
cdef extern from "list_tuple_slices.h":
inline object __Pyx_PyList_GetSlice(object ob, int start, int stop)
inline object __Pyx_PyTuple_GetSlice(object ob, int start, int stop)
cdef list lst = list(range(10))
cdef list lst2 = list(range(7))
def get_slice1(list lst):
cdef int i
cdef list res = []
for i in range(20):
res.append(PySequence_GetSlice(lst, 2, 8))
return res
def get_slice2(list lst):
cdef int i
cdef list res = []
for i in range(20):
res.append(__Pyx_PyList_GetSlice(lst, 2, 8))
return res
def test_get_slice1():
get_slice1(lst)
def test_get_slice2():
get_slice2(lst)
tuple_slicing.pyx
---
from cpython.sequence cimport PySequence_GetSlice
cdef extern from "list_tuple_slices.h":
inline object __Pyx_PyList_GetSlice(object lst, int start, int stop)
inline object __Pyx_PyTuple_GetSlice(object ob, int start, int stop)
cdef tuple lst = tuple(range(10))
def get_slice1(tuple lst):
cdef int i
cdef list res = []
for i in range(20):
res.append(PySequence_GetSlice(lst, 2, 8))
return res
def get_slice2(tuple lst):
cdef int i
cdef list res = []
for i in range(20):
res.append(__Pyx_PyTuple_GetSlice(lst, 2, 8))
return res
def test_get_slice1():
get_slice1(lst)
def test_get_slice2():
get_slice2(lst)
Here are timings:
for list
(py33) zbook:mytests $ python -m timeit -n 100 -r 5 -v -s "from
mytests.list_slice import test_get_slice1" "test_get_slice1()"
raw times: 10.2 10.3 10.4 10.1 10.2
100 loops, best of 5: 101 msec per loop
(py33) zbook:mytests $ python -m timeit -n 100 -r 5 -v -s "from
mytests.list_slice import test_get_slice1" "test_get_slice1()"
raw times: 10.3 10.3 10.2 10.3 10.2
100 loops, best of 5: 102 msec per loop
(py33) zbook:mytests $ python -m timeit -n 100 -r 5 -v -s "from
mytests.list_slice import test_get_slice2" "test_get_slice2()"
raw times: 8.16 8.19 8.17 8.2 8.16
100 loops
