Re: [Cython] Cython 0.17 beta 3 released - release candidate

[Christoph Gohlke]

On 8/25/2012 6:51 AM, Stefan Behnel wrote:

Christoph Gohlke, 25.08.2012 04:07:

On 8/24/2012 12:43 PM, Stefan Behnel wrote:

Christoph Gohlke, 24.08.2012 07:20:

I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers.

32 bit Python 2.7 works well, only 4 test failures.


Three of those errors are in OpenMP tests - is OpenMP supported in your
build environment?


OpenMP is available on my system, and parallel.pyd is linked to the openmp
library. The prange tests fail only sometimes. On my system, the prange
index is sometimes left at the start (zero) of the range, while the tests
expect the index to be left at the stop of the range. According to the
Cython prange enhancements webpage "the iterations of the loop body can be
executed in any order" . Where
does that leave the loop index?


Mark should know best. I agree that this seems a bit smelly.



The other one is the new "initial_file_path" test that fails with this
linker error:

"""
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL
/nologo /INCREMENTAL:NO /INCREMENTAL:YES/DEBUG /LIBPATH:X:\Python27\libs
/LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__
build\temp.win32-2.7\Release\my_test_package\__init__.obj
/OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd
/IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib
/MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest


LINK : error LNK2001: unresolved external symbol init__init__
"""

Maybe the Windows build of distutils is broken here - it seems to assume
the wrong module name for the package module.


I think this is an issue with the test. The extension does compile and link
outside of the tests.


Hmm, did you run the setup.py script? It compiles the "__init__.py" file as
well, which is the problem above. Could you check if that's properly being
imported as a shared library?

(The way "somefile.srctree" tests work is that they explode into a
directory structure and then just run the commands in the first lines.)



On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes during
`test_slice_assignment (memslice.__test__)`. I tested two computers. The
Windows executive can not identify in which specific module it crashes, and
neither enabling faulthandler nor building with debug symbols gives any
useful information. Can anyone reproduce this? It seems compiler specific
since Python 3.3, which is using msvc10, does not crash.


Hmm, yes, sounds like a problem with the compiler. Would be good to get
this sorted out, but it's almost impossible to debug something like this
from a distance.


Maybe the following simple example is related. It fails (not crash) when
compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32
and 64 bit):

```
from cython.view cimport array as cvarray
import numpy as np

narr = np.arange(8 * 14 * 11).reshape((8, 14, 11))

cdef int[:, :, ::1] a = narr
cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3]

print narr[2:8:2, -4:1:-1, 1:3].shape
print b.shape[0], b.shape[1], b.shape[2]
```

On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2)


I'll leave that for the others to comment.



This failure and the test crash disappear when auto-inlining is 
disabled, e.g. using the `/Ob1` compiler switch 



Is there a way to add a C pre-processor `#pragma` to a Cython generated 
C file? I tried `cdef void emit_pragma '#pragma auto_inline(off) //' ()` 
but the pragma ends up inside the Python module init function, which 
fails to compile with `error C2156: pragma must be outside function`.







"""
bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified
multiple times; using first specification
"""


This is "normal".


So you have an idea why this happens? Is it because of that explicit
/EXPORT command line switch for the module init function name?


seems to be an issue with distutils: 

Christoph




Stefan

___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel




___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[Stefan Behnel]

Christoph Gohlke, 26.08.2012 10:03:
> On 8/25/2012 6:51 AM, Stefan Behnel wrote:
>> Christoph Gohlke, 25.08.2012 04:07:
>>> On 8/24/2012 12:43 PM, Stefan Behnel wrote:
 Christoph Gohlke, 24.08.2012 07:20:
> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes
> during
> `test_slice_assignment (memslice.__test__)`. I tested two computers. The
> Windows executive can not identify in which specific module it
> crashes, and
> neither enabling faulthandler nor building with debug symbols gives any
> useful information. Can anyone reproduce this? It seems compiler specific
> since Python 3.3, which is using msvc10, does not crash.

 Hmm, yes, sounds like a problem with the compiler. Would be good to get
 this sorted out, but it's almost impossible to debug something like this
 from a distance.
>>>
>>> Maybe the following simple example is related. It fails (not crash) when
>>> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32
>>> and 64 bit):
>>>
>>> ```
>>> from cython.view cimport array as cvarray
>>> import numpy as np
>>>
>>> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11))
>>>
>>> cdef int[:, :, ::1] a = narr
>>> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3]
>>>
>>> print narr[2:8:2, -4:1:-1, 1:3].shape
>>> print b.shape[0], b.shape[1], b.shape[2]
>>> ```
>>>
>>> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2)
>>
>> I'll leave that for the others to comment.
> 
> This failure and the test crash disappear when auto-inlining is disabled,
> e.g. using the `/Ob1` compiler switch
> 

Thanks for figuring that out.


> Is there a way to add a C pre-processor `#pragma` to a Cython generated C
> file?

I don't think so, not in arbitrary places.


> I tried `cdef void emit_pragma '#pragma auto_inline(off) //' ()` but
> the pragma ends up inside the Python module init function, which fails to
> compile with `error C2156: pragma must be outside function`.

As a quick fix, we could wrap (or just start) our utility code section with
that pragma. It starts after all user code. According to the docs, the
pragma only disables automatic inlining in the specific range, not
explicitly requested inlining.

http://msdn.microsoft.com/en-us/library/ah08w5c3%28v=vs.71%29.aspx

We usually know what we consider worth inlining and what doesn't need to
be. If we can find out what exact functions trigger this, we can start
being more specific about where to enable the pragma, but I think it's ok
to enable it on all utility code for now.

Could you figure out a suitable preprocessor guard that only enables it for
the affected compilers?

Stefan

___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[Stefan Behnel]

Stefan Behnel, 26.08.2012 11:36:
> Christoph Gohlke, 26.08.2012 10:03:
>> On 8/25/2012 6:51 AM, Stefan Behnel wrote:
>>> Christoph Gohlke, 25.08.2012 04:07:
 On 8/24/2012 12:43 PM, Stefan Behnel wrote:
> Christoph Gohlke, 24.08.2012 07:20:
>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes
>> during
>> `test_slice_assignment (memslice.__test__)`. I tested two computers. The
>> Windows executive can not identify in which specific module it
>> crashes, and
>> neither enabling faulthandler nor building with debug symbols gives any
>> useful information. Can anyone reproduce this? It seems compiler specific
>> since Python 3.3, which is using msvc10, does not crash.
>
> Hmm, yes, sounds like a problem with the compiler. Would be good to get
> this sorted out, but it's almost impossible to debug something like this
> from a distance.

 Maybe the following simple example is related. It fails (not crash) when
 compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32
 and 64 bit):

 ```
 from cython.view cimport array as cvarray
 import numpy as np

 narr = np.arange(8 * 14 * 11).reshape((8, 14, 11))

 cdef int[:, :, ::1] a = narr
 cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3]

 print narr[2:8:2, -4:1:-1, 1:3].shape
 print b.shape[0], b.shape[1], b.shape[2]
 ```

 On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2)
>>>
>>> I'll leave that for the others to comment.
>>
>> This failure and the test crash disappear when auto-inlining is disabled,
>> e.g. using the `/Ob1` compiler switch
>> 
> 
> Thanks for figuring that out.
> 
> 
>> Is there a way to add a C pre-processor `#pragma` to a Cython generated C
>> file?
> 
> I don't think so, not in arbitrary places.
> 
> 
>> I tried `cdef void emit_pragma '#pragma auto_inline(off) //' ()` but
>> the pragma ends up inside the Python module init function, which fails to
>> compile with `error C2156: pragma must be outside function`.
> 
> As a quick fix, we could wrap (or just start) our utility code section with
> that pragma. It starts after all user code. According to the docs, the
> pragma only disables automatic inlining in the specific range, not
> explicitly requested inlining.
> 
> http://msdn.microsoft.com/en-us/library/ah08w5c3%28v=vs.71%29.aspx
> 
> We usually know what we consider worth inlining and what doesn't need to
> be. If we can find out what exact functions trigger this, we can start
> being more specific about where to enable the pragma, but I think it's ok
> to enable it on all utility code for now.
> 
> Could you figure out a suitable preprocessor guard that only enables it for
> the affected compilers?

A patch would essentially look like this:

diff -r 18fed0dec20e Cython/Compiler/Code.py
--- a/Cython/Compiler/Code.py   Sun Aug 26 00:54:01 2012 +0200
+++ b/Cython/Compiler/Code.py   Sun Aug 26 11:39:29 2012 +0200
@@ -893,6 +893,8 @@
 code.write('\n#line 1 "cython_utility"\n')
 code.putln("")
 code.putln("/* Runtime support code */")
+code.putln("#pragma auto_inline(off)")
+code.putln("")

 def finalize_main_c_code(self):
 self.close_global_decls()

Stefan

___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[mark florisson]

On 25 August 2012 03:07, Christoph Gohlke  wrote:
> Hi,
>
>
> On 8/24/2012 12:43 PM, Stefan Behnel wrote:
>>
>> Hi,
>>
>> thanks for testing!
>>
>> Christoph Gohlke, 24.08.2012 07:20:
>>>
>>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers.
>>>
>>> 32 bit Python 2.7 works well, only 4 test failures.
>>
>>
>> Three of those errors are in OpenMP tests - is OpenMP supported in your
>> build environment?
>
>
> OpenMP is available on my system, and parallel.pyd is linked to the openmp
> library. The prange tests fail only sometimes. On my system, the prange
> index is sometimes left at the start (zero) of the range, while the tests
> expect the index to be left at the stop of the range. According to the
> Cython prange enhancements webpage "the iterations of the loop body can be
> executed in any order" . Where
> does that leave the loop index?
>

I should be the value from the last iteration, but in my experience
many compilers have buggy OpenMP implementations. I think your
compiler doesn't correctly support the lastprivate clause in all
situations. For instance, test_prange fails (which doesn't have any
break/return/exceptions), which simply has a lastprivate clause and a
schedule clause set to 'dynamic'. The test without the schedule clause
works fine (or maybe it's just luck). Or maybe it doesn't support
multiple lastprivate() clauses? I'm not entirely sure... It also seems
the thread limit on your system is 1.

In any case, the generated code for these tests looks correct to me,
but we've had similar problems before with different compilers...

>
>>
>> The other one is the new "initial_file_path" test that fails with this
>> linker error:
>>
>> """
>> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL
>> /nologo /INCREMENTAL:NO /INCREMENTAL:YES/DEBUG
>> /LIBPATH:X:\Python27\libs
>> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__
>> build\temp.win32-2.7\Release\my_test_package\__init__.obj
>>
>> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd
>> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib
>>
>> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest
>>
>> LINK : error LNK2001: unresolved external symbol init__init__
>> """
>>
>> Maybe the Windows build of distutils is broken here - it seems to assume
>> the wrong module name for the package module.
>
>
> I think this is an issue with the test. The extension does compile and link
> outside of the tests.
>
>
>
>>
>> I guess compiling package modules is just an overall badly supported
>> feature in CPython...
>>
>>
>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes
>>> during
>>> `test_slice_assignment (memslice.__test__)`. I tested two computers. The
>>> Windows executive can not identify in which specific module it crashes,
>>> and
>>> neither enabling faulthandler nor building with debug symbols gives any
>>> useful information. Can anyone reproduce this? It seems compiler specific
>>> since Python 3.3, which is using msvc10, does not crash.
>>
>>
>> Hmm, yes, sounds like a problem with the compiler. Would be good to get
>> this sorted out, but it's almost impossible to debug something like this
>> from a distance.
>
>
>
> Maybe the following simple example is related. It fails (not crash) when
> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32
> and 64 bit):
>
> ```
> from cython.view cimport array as cvarray
> import numpy as np
>
> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11))
>
> cdef int[:, :, ::1] a = narr
> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3]
>
> print narr[2:8:2, -4:1:-1, 1:3].shape
> print b.shape[0], b.shape[1], b.shape[2]
> ```
>
> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2)
>
>
>
>>
>>
>>> When disabling
>>> test_slice_assignment, runtests.py completes with many failures.
>>>
>>> The results of `runtests.py -v -v` are at
>>> .
>>
>>
>> The 64bit output looks so broken that I wonder what went wrong here. I
>> mean, most of the problems look like this:
>>
>> """
>> Expected:
>>  Traceback (most recent call last):
>>  TypeError: m() takes at most 2 positional arguments (3 given)
>> Got:
>>  Traceback (most recent call last):
>>  TypeError: m() takes at most %Id positional argument%s (%Id given)
>> """
>>
>> I have no idea how that can happen.
>>
>> I can see two other problems, one is the linker warning about the module
>> init function in Py3:
>>
>> """
>> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified
>> multiple times; using first specification
>> """
>
>
> This is "normal".
>
>
>
>>
>> The other one is about mixing Py_ssize_t and int (and some other) types
>> all
>> over the place:
>>
>> """
>> bufaccess.c(3033) : warning C4244: '=' : conversion from 'Py_ssize_t' to
>> 'int', possible loss of da

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[Christoph Gohlke]

On 8/26/2012 2:41 AM, Stefan Behnel wrote:

Stefan Behnel, 26.08.2012 11:36:

Christoph Gohlke, 26.08.2012 10:03:

On 8/25/2012 6:51 AM, Stefan Behnel wrote:

Christoph Gohlke, 25.08.2012 04:07:

On 8/24/2012 12:43 PM, Stefan Behnel wrote:

Christoph Gohlke, 24.08.2012 07:20:

On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes
during
`test_slice_assignment (memslice.__test__)`. I tested two computers. The
Windows executive can not identify in which specific module it
crashes, and
neither enabling faulthandler nor building with debug symbols gives any
useful information. Can anyone reproduce this? It seems compiler specific
since Python 3.3, which is using msvc10, does not crash.


Hmm, yes, sounds like a problem with the compiler. Would be good to get
this sorted out, but it's almost impossible to debug something like this
from a distance.


Maybe the following simple example is related. It fails (not crash) when
compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32
and 64 bit):

```
from cython.view cimport array as cvarray
import numpy as np

narr = np.arange(8 * 14 * 11).reshape((8, 14, 11))

cdef int[:, :, ::1] a = narr
cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3]

print narr[2:8:2, -4:1:-1, 1:3].shape
print b.shape[0], b.shape[1], b.shape[2]
```

On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2)


I'll leave that for the others to comment.


This failure and the test crash disappear when auto-inlining is disabled,
e.g. using the `/Ob1` compiler switch



Thanks for figuring that out.



Is there a way to add a C pre-processor `#pragma` to a Cython generated C
file?


I don't think so, not in arbitrary places.



I tried `cdef void emit_pragma '#pragma auto_inline(off) //' ()` but
the pragma ends up inside the Python module init function, which fails to
compile with `error C2156: pragma must be outside function`.


As a quick fix, we could wrap (or just start) our utility code section with
that pragma. It starts after all user code. According to the docs, the
pragma only disables automatic inlining in the specific range, not
explicitly requested inlining.

http://msdn.microsoft.com/en-us/library/ah08w5c3%28v=vs.71%29.aspx

We usually know what we consider worth inlining and what doesn't need to
be. If we can find out what exact functions trigger this, we can start
being more specific about where to enable the pragma, but I think it's ok
to enable it on all utility code for now.

Could you figure out a suitable preprocessor guard that only enables it for
the affected compilers?


How about `#if defined(_WIN64) && (_MSC_VER == 1500)`




A patch would essentially look like this:

diff -r 18fed0dec20e Cython/Compiler/Code.py
--- a/Cython/Compiler/Code.py   Sun Aug 26 00:54:01 2012 +0200
+++ b/Cython/Compiler/Code.py   Sun Aug 26 11:39:29 2012 +0200
@@ -893,6 +893,8 @@
  code.write('\n#line 1 "cython_utility"\n')
  code.putln("")
  code.putln("/* Runtime support code */")
+code.putln("#pragma auto_inline(off)")
+code.putln("")

  def finalize_main_c_code(self):
  self.close_global_decls()

Stefan



Thanks. I just tried this. The `#pragma auto_inline(off)` line appears 
after the MemoryView code and has no effect on that part.


Christoph

___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[Christoph Gohlke]

On 8/26/2012 4:08 AM, mark florisson wrote:

On 25 August 2012 03:07, Christoph Gohlke  wrote:

Hi,


On 8/24/2012 12:43 PM, Stefan Behnel wrote:


Hi,

thanks for testing!

Christoph Gohlke, 24.08.2012 07:20:


I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers.

32 bit Python 2.7 works well, only 4 test failures.



Three of those errors are in OpenMP tests - is OpenMP supported in your
build environment?



OpenMP is available on my system, and parallel.pyd is linked to the openmp
library. The prange tests fail only sometimes. On my system, the prange
index is sometimes left at the start (zero) of the range, while the tests
expect the index to be left at the stop of the range. According to the
Cython prange enhancements webpage "the iterations of the loop body can be
executed in any order" . Where
does that leave the loop index?



I should be the value from the last iteration, but in my experience
many compilers have buggy OpenMP implementations. I think your
compiler doesn't correctly support the lastprivate clause in all
situations. For instance, test_prange fails (which doesn't have any
break/return/exceptions), which simply has a lastprivate clause and a
schedule clause set to 'dynamic'. The test without the schedule clause
works fine (or maybe it's just luck). Or maybe it doesn't support
multiple lastprivate() clauses? I'm not entirely sure... It also seems
the thread limit on your system is 1.


Thanks. Good to know.

Regarding the thread limit of 1: according to 
, 
if omp_get_num_threads is "called from a serial portion of a program, or 
from a nested parallel region that is serialized, this function returns 
1." This seems to be the case here (looking at the generated C code for 
the test_num_threads function).


In the example at 
 
they add a `#pragma omp master` directive before calling 
omp_get_num_threads( ).


Christoph




In any case, the generated code for these tests looks correct to me,
but we've had similar problems before with different compilers...





The other one is the new "initial_file_path" test that fails with this
linker error:

"""
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL
/nologo /INCREMENTAL:NO /INCREMENTAL:YES/DEBUG
/LIBPATH:X:\Python27\libs
/LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__
build\temp.win32-2.7\Release\my_test_package\__init__.obj

/OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd
/IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib

/MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest

LINK : error LNK2001: unresolved external symbol init__init__
"""

Maybe the Windows build of distutils is broken here - it seems to assume
the wrong module name for the package module.



I think this is an issue with the test. The extension does compile and link
outside of the tests.





I guess compiling package modules is just an overall badly supported
feature in CPython...



On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes
during
`test_slice_assignment (memslice.__test__)`. I tested two computers. The
Windows executive can not identify in which specific module it crashes,
and
neither enabling faulthandler nor building with debug symbols gives any
useful information. Can anyone reproduce this? It seems compiler specific
since Python 3.3, which is using msvc10, does not crash.



Hmm, yes, sounds like a problem with the compiler. Would be good to get
this sorted out, but it's almost impossible to debug something like this
from a distance.




Maybe the following simple example is related. It fails (not crash) when
compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32
and 64 bit):

```
from cython.view cimport array as cvarray
import numpy as np

narr = np.arange(8 * 14 * 11).reshape((8, 14, 11))

cdef int[:, :, ::1] a = narr
cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3]

print narr[2:8:2, -4:1:-1, 1:3].shape
print b.shape[0], b.shape[1], b.shape[2]
```

On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2)







When disabling
test_slice_assignment, runtests.py completes with many failures.

The results of `runtests.py -v -v` are at
.



The 64bit output looks so broken that I wonder what went wrong here. I
mean, most of the problems look like this:

"""
Expected:
  Traceback (most recent call last):
  TypeError: m() takes at most 2 positional arguments (3 given)
Got:
  Traceback (most recent call last):
  TypeError: m() takes at most %Id positional argument%s (%Id given)
"""

I have no idea how that can happen.

I can see two other problems, one is the linker warning about the module
init function in Py3:

"""
bufaccess.obj : warnin

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[Stefan Behnel]

Christoph Gohlke, 26.08.2012 17:54:
> On 8/26/2012 2:41 AM, Stefan Behnel wrote:
>> A patch would essentially look like this:
>>
>> diff -r 18fed0dec20e Cython/Compiler/Code.py
>> --- a/Cython/Compiler/Code.py   Sun Aug 26 00:54:01 2012 +0200
>> +++ b/Cython/Compiler/Code.py   Sun Aug 26 11:39:29 2012 +0200
>> @@ -893,6 +893,8 @@
>>   code.write('\n#line 1 "cython_utility"\n')
>>   code.putln("")
>>   code.putln("/* Runtime support code */")
>> +code.putln("#pragma auto_inline(off)")
>> +code.putln("")
>>
>>   def finalize_main_c_code(self):
>>   self.close_global_decls()
> 
> Thanks. I just tried this. The `#pragma auto_inline(off)` line appears
> after the MemoryView code and has no effect on that part.

Argh. That's too bad - the memory view code is actually compiled from
Cython code and gets injected at the tree level, not by the code generator.

I would like to avoid an impact on user code by this pragma, so I think
we'll have to find a way to inject the pragma into the code stream somehow...

Stefan

___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[mark florisson]

On 26 August 2012 17:32, Christoph Gohlke  wrote:
> On 8/26/2012 4:08 AM, mark florisson wrote:
>>
>> On 25 August 2012 03:07, Christoph Gohlke  wrote:
>>>
>>> Hi,
>>>
>>>
>>> On 8/24/2012 12:43 PM, Stefan Behnel wrote:


 Hi,

 thanks for testing!

 Christoph Gohlke, 24.08.2012 07:20:
>
>
> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers.
>
> 32 bit Python 2.7 works well, only 4 test failures.



 Three of those errors are in OpenMP tests - is OpenMP supported in your
 build environment?
>>>
>>>
>>>
>>> OpenMP is available on my system, and parallel.pyd is linked to the
>>> openmp
>>> library. The prange tests fail only sometimes. On my system, the prange
>>> index is sometimes left at the start (zero) of the range, while the tests
>>> expect the index to be left at the stop of the range. According to the
>>> Cython prange enhancements webpage "the iterations of the loop body can
>>> be
>>> executed in any order" .
>>> Where
>>> does that leave the loop index?
>>>
>>
>> I should be the value from the last iteration, but in my experience
>> many compilers have buggy OpenMP implementations. I think your
>> compiler doesn't correctly support the lastprivate clause in all
>> situations. For instance, test_prange fails (which doesn't have any
>> break/return/exceptions), which simply has a lastprivate clause and a
>> schedule clause set to 'dynamic'. The test without the schedule clause
>> works fine (or maybe it's just luck). Or maybe it doesn't support
>> multiple lastprivate() clauses? I'm not entirely sure... It also seems
>> the thread limit on your system is 1.
>
>
> Thanks. Good to know.
>
> Regarding the thread limit of 1: according to
> , if
> omp_get_num_threads is "called from a serial portion of a program, or from a
> nested parallel region that is serialized, this function returns 1." This
> seems to be the case here (looking at the generated C code for the
> test_num_threads function).

I don't think so, all calls are made from within parallel sections in
the test code, and the corresponding generated code. But OpenMP
implementations are allowed to give you less threads if the
implementation doesn't support more, or if the OS doesn't allow more.
In any case, that test is kind of a non-problem.

> In the example at
>  they
> add a `#pragma omp master` directive before calling omp_get_num_threads( ).

That's because they only need one assignment and call.

> Christoph
>
>
>
>>
>> In any case, the generated code for these tests looks correct to me,
>> but we've had similar problems before with different compilers...
>>
>>>

 The other one is the new "initial_file_path" test that fails with this
 linker error:

 """
 C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL
 /nologo /INCREMENTAL:NO /INCREMENTAL:YES/DEBUG
 /LIBPATH:X:\Python27\libs
 /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__
 build\temp.win32-2.7\Release\my_test_package\__init__.obj


 /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd
 /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib


 /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest

 LINK : error LNK2001: unresolved external symbol init__init__
 """

 Maybe the Windows build of distutils is broken here - it seems to assume
 the wrong module name for the package module.
>>>
>>>
>>>
>>> I think this is an issue with the test. The extension does compile and
>>> link
>>> outside of the tests.
>>>
>>>
>>>

 I guess compiling package modules is just an overall badly supported
 feature in CPython...


> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes
> during
> `test_slice_assignment (memslice.__test__)`. I tested two computers.
> The
> Windows executive can not identify in which specific module it crashes,
> and
> neither enabling faulthandler nor building with debug symbols gives any
> useful information. Can anyone reproduce this? It seems compiler
> specific
> since Python 3.3, which is using msvc10, does not crash.



 Hmm, yes, sounds like a problem with the compiler. Would be good to get
 this sorted out, but it's almost impossible to debug something like this
 from a distance.
>>>
>>>
>>>
>>>
>>> Maybe the following simple example is related. It fails (not crash) when
>>> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10
>>> (32
>>> and 64 bit):
>>>
>>> ```
>>> from cython.view cimport array as cvarray
>>> import numpy as np
>>>
>>> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11))
>>>
>>> cdef int[:,

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[Christoph Gohlke]

On 8/26/2012 11:08 AM, mark florisson wrote:

On 26 August 2012 17:32, Christoph Gohlke  wrote:

On 8/26/2012 4:08 AM, mark florisson wrote:


On 25 August 2012 03:07, Christoph Gohlke  wrote:


Hi,


On 8/24/2012 12:43 PM, Stefan Behnel wrote:



Hi,

thanks for testing!

Christoph Gohlke, 24.08.2012 07:20:



I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers.

32 bit Python 2.7 works well, only 4 test failures.




Three of those errors are in OpenMP tests - is OpenMP supported in your
build environment?




OpenMP is available on my system, and parallel.pyd is linked to the
openmp
library. The prange tests fail only sometimes. On my system, the prange
index is sometimes left at the start (zero) of the range, while the tests
expect the index to be left at the stop of the range. According to the
Cython prange enhancements webpage "the iterations of the loop body can
be
executed in any order" .
Where
does that leave the loop index?



I should be the value from the last iteration, but in my experience
many compilers have buggy OpenMP implementations. I think your
compiler doesn't correctly support the lastprivate clause in all
situations. For instance, test_prange fails (which doesn't have any
break/return/exceptions), which simply has a lastprivate clause and a
schedule clause set to 'dynamic'. The test without the schedule clause
works fine (or maybe it's just luck). Or maybe it doesn't support
multiple lastprivate() clauses? I'm not entirely sure... It also seems
the thread limit on your system is 1.



Thanks. Good to know.

Regarding the thread limit of 1: according to
, if
omp_get_num_threads is "called from a serial portion of a program, or from a
nested parallel region that is serialized, this function returns 1." This
seems to be the case here (looking at the generated C code for the
test_num_threads function).


I don't think so, all calls are made from within parallel sections in
the test code, and the corresponding generated code. But OpenMP
implementations are allowed to give you less threads if the
implementation doesn't support more, or if the OS doesn't allow more.
In any case, that test is kind of a non-problem.


In the example at
 they
add a `#pragma omp master` directive before calling omp_get_num_threads( ).


That's because they only need one assignment and call.



You are right of course. Sorry. I verified that with a simpler example.
It seems that runtests.py does not add the `/openmp` flag to the 
compiler arguments when compiling C++ code. If I change line 119 in 
runtests.py to return `'/openmp', ''` instead of None the 
test_num_threads failure disappears. But the cpp\parallel\prange tests 
start to fail too.




Christoph








In any case, the generated code for these tests looks correct to me,
but we've had similar problems before with different compilers...





The other one is the new "initial_file_path" test that fails with this
linker error:

"""
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL
/nologo /INCREMENTAL:NO /INCREMENTAL:YES/DEBUG
/LIBPATH:X:\Python27\libs
/LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__
build\temp.win32-2.7\Release\my_test_package\__init__.obj


/OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd
/IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib


/MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest

LINK : error LNK2001: unresolved external symbol init__init__
"""

Maybe the Windows build of distutils is broken here - it seems to assume
the wrong module name for the package module.




I think this is an issue with the test. The extension does compile and
link
outside of the tests.





I guess compiling package modules is just an overall badly supported
feature in CPython...



On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes
during
`test_slice_assignment (memslice.__test__)`. I tested two computers.
The
Windows executive can not identify in which specific module it crashes,
and
neither enabling faulthandler nor building with debug symbols gives any
useful information. Can anyone reproduce this? It seems compiler
specific
since Python 3.3, which is using msvc10, does not crash.




Hmm, yes, sounds like a problem with the compiler. Would be good to get
this sorted out, but it's almost impossible to debug something like this
from a distance.





Maybe the following simple example is related. It fails (not crash) when
compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10
(32
and 64 bit):

```
from cython.view cimport array as cvarray
import numpy as np

narr = np.arange(8 * 14 * 11).reshape((8, 14, 11))

cdef int[:, :, ::1] a = narr
cdef int[:, :, 

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[Christoph Gohlke]

On 8/26/2012 4:08 AM, mark florisson wrote:

On 25 August 2012 03:07, Christoph Gohlke  wrote:

Hi,


On 8/24/2012 12:43 PM, Stefan Behnel wrote:


Hi,

thanks for testing!

Christoph Gohlke, 24.08.2012 07:20:


I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers.

32 bit Python 2.7 works well, only 4 test failures.



Three of those errors are in OpenMP tests - is OpenMP supported in your
build environment?



OpenMP is available on my system, and parallel.pyd is linked to the openmp
library. The prange tests fail only sometimes. On my system, the prange
index is sometimes left at the start (zero) of the range, while the tests
expect the index to be left at the stop of the range. According to the
Cython prange enhancements webpage "the iterations of the loop body can be
executed in any order" . Where
does that leave the loop index?



I should be the value from the last iteration, but in my experience
many compilers have buggy OpenMP implementations. I think your
compiler doesn't correctly support the lastprivate clause in all
situations. For instance, test_prange fails (which doesn't have any
break/return/exceptions), which simply has a lastprivate clause and a
schedule clause set to 'dynamic'. The test without the schedule clause
works fine (or maybe it's just luck). Or maybe it doesn't support
multiple lastprivate() clauses? I'm not entirely sure... It also seems
the thread limit on your system is 1.

In any case, the generated code for these tests looks correct to me,
but we've had similar problems before with different compilers...


A minimal example that fails for me is:

def test_parallel():
cdef int i = 0, s = 0
with nogil, cython.parallel.parallel():
for i in prange(10):
s += i
return i

The returned value is often 0, otherwise 9 as expected.

In the generated C code I see

#pragma omp for firstprivate(__pyx_v_i) lastprivate(__pyx_v_i)

If I change this to

#pragma omp parallel for firstprivate(__pyx_v_i) lastprivate(__pyx_v_i)

the function always returns the expected value. Does that make sense?

Thanks for your help and patience.

Christoph








The other one is the new "initial_file_path" test that fails with this
linker error:

"""
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL
/nologo /INCREMENTAL:NO /INCREMENTAL:YES/DEBUG
/LIBPATH:X:\Python27\libs
/LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__
build\temp.win32-2.7\Release\my_test_package\__init__.obj

/OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd
/IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib

/MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest

LINK : error LNK2001: unresolved external symbol init__init__
"""

Maybe the Windows build of distutils is broken here - it seems to assume
the wrong module name for the package module.



I think this is an issue with the test. The extension does compile and link
outside of the tests.





I guess compiling package modules is just an overall badly supported
feature in CPython...



On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes
during
`test_slice_assignment (memslice.__test__)`. I tested two computers. The
Windows executive can not identify in which specific module it crashes,
and
neither enabling faulthandler nor building with debug symbols gives any
useful information. Can anyone reproduce this? It seems compiler specific
since Python 3.3, which is using msvc10, does not crash.



Hmm, yes, sounds like a problem with the compiler. Would be good to get
this sorted out, but it's almost impossible to debug something like this
from a distance.




Maybe the following simple example is related. It fails (not crash) when
compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32
and 64 bit):

```
from cython.view cimport array as cvarray
import numpy as np

narr = np.arange(8 * 14 * 11).reshape((8, 14, 11))

cdef int[:, :, ::1] a = narr
cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3]

print narr[2:8:2, -4:1:-1, 1:3].shape
print b.shape[0], b.shape[1], b.shape[2]
```

On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2)







When disabling
test_slice_assignment, runtests.py completes with many failures.

The results of `runtests.py -v -v` are at
.



The 64bit output looks so broken that I wonder what went wrong here. I
mean, most of the problems look like this:

"""
Expected:
  Traceback (most recent call last):
  TypeError: m() takes at most 2 positional arguments (3 given)
Got:
  Traceback (most recent call last):
  TypeError: m() takes at most %Id positional argument%s (%Id given)
"""

I have no idea how that can happen.

I can see two other problems, one is the linker warning about the module
init function in Py3:

"""
bufaccess.obj : warning LNK4197: expo

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[Christoph Gohlke]

On 8/26/2012 2:09 PM, Christoph Gohlke wrote:

On 8/26/2012 4:08 AM, mark florisson wrote:

On 25 August 2012 03:07, Christoph Gohlke  wrote:

Hi,


On 8/24/2012 12:43 PM, Stefan Behnel wrote:


Hi,

thanks for testing!

Christoph Gohlke, 24.08.2012 07:20:


I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers.

32 bit Python 2.7 works well, only 4 test failures.



Three of those errors are in OpenMP tests - is OpenMP supported in your
build environment?



OpenMP is available on my system, and parallel.pyd is linked to the
openmp
library. The prange tests fail only sometimes. On my system, the prange
index is sometimes left at the start (zero) of the range, while the
tests
expect the index to be left at the stop of the range. According to the
Cython prange enhancements webpage "the iterations of the loop body
can be
executed in any order" .
Where
does that leave the loop index?



I should be the value from the last iteration, but in my experience
many compilers have buggy OpenMP implementations. I think your
compiler doesn't correctly support the lastprivate clause in all
situations. For instance, test_prange fails (which doesn't have any
break/return/exceptions), which simply has a lastprivate clause and a
schedule clause set to 'dynamic'. The test without the schedule clause
works fine (or maybe it's just luck). Or maybe it doesn't support
multiple lastprivate() clauses? I'm not entirely sure... It also seems
the thread limit on your system is 1.

In any case, the generated code for these tests looks correct to me,
but we've had similar problems before with different compilers...


A minimal example that fails for me is:

def test_parallel():
 cdef int i = 0, s = 0
 with nogil, cython.parallel.parallel():
 for i in prange(10):
 s += i
 return i

The returned value is often 0, otherwise 9 as expected.

In the generated C code I see

 #pragma omp for firstprivate(__pyx_v_i) lastprivate(__pyx_v_i)

If I change this to

 #pragma omp parallel for firstprivate(__pyx_v_i)
lastprivate(__pyx_v_i)

the function always returns the expected value. Does that make sense?



Well, apparently it doesn't make sense because the value of s is not 
correct.


Christoph




Thanks for your help and patience.

Christoph








The other one is the new "initial_file_path" test that fails with this
linker error:

"""
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL
/nologo /INCREMENTAL:NO /INCREMENTAL:YES/DEBUG
/LIBPATH:X:\Python27\libs
/LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__
build\temp.win32-2.7\Release\my_test_package\__init__.obj

/OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd

/IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib

/MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest


LINK : error LNK2001: unresolved external symbol init__init__
"""

Maybe the Windows build of distutils is broken here - it seems to
assume
the wrong module name for the package module.



I think this is an issue with the test. The extension does compile
and link
outside of the tests.





I guess compiling package modules is just an overall badly supported
feature in CPython...



On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes
during
`test_slice_assignment (memslice.__test__)`. I tested two
computers. The
Windows executive can not identify in which specific module it
crashes,
and
neither enabling faulthandler nor building with debug symbols gives
any
useful information. Can anyone reproduce this? It seems compiler
specific
since Python 3.3, which is using msvc10, does not crash.



Hmm, yes, sounds like a problem with the compiler. Would be good to get
this sorted out, but it's almost impossible to debug something like
this
from a distance.




Maybe the following simple example is related. It fails (not crash) when
compiled with 64 bit msvc9, but does work with 32 bit msvc9 and
msvc10 (32
and 64 bit):

```
from cython.view cimport array as cvarray
import numpy as np

narr = np.arange(8 * 14 * 11).reshape((8, 14, 11))

cdef int[:, :, ::1] a = narr
cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3]

print narr[2:8:2, -4:1:-1, 1:3].shape
print b.shape[0], b.shape[1], b.shape[2]
```

On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3,
9, 2)







When disabling
test_slice_assignment, runtests.py completes with many failures.

The results of `runtests.py -v -v` are at
.



The 64bit output looks so broken that I wonder what went wrong here. I
mean, most of the problems look like this:

"""
Expected:
  Traceback (most recent call last):
  TypeError: m() takes at most 2 positional arguments (3 given)
Got:
  Traceback (most recent call last):
  TypeError: m() takes at most %Id positional argument%s (%Id
given)
"""

I have no idea how th

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[mark florisson]

On 26 August 2012 22:45, mark florisson  wrote:
> On 26 August 2012 22:25, Christoph Gohlke  wrote:
>> On 8/26/2012 2:09 PM, Christoph Gohlke wrote:
>>>
>>> On 8/26/2012 4:08 AM, mark florisson wrote:

 On 25 August 2012 03:07, Christoph Gohlke  wrote:
>
> Hi,
>
>
> On 8/24/2012 12:43 PM, Stefan Behnel wrote:
>>
>>
>> Hi,
>>
>> thanks for testing!
>>
>> Christoph Gohlke, 24.08.2012 07:20:
>>>
>>>
>>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers.
>>>
>>> 32 bit Python 2.7 works well, only 4 test failures.
>>
>>
>>
>> Three of those errors are in OpenMP tests - is OpenMP supported in your
>> build environment?
>
>
>
> OpenMP is available on my system, and parallel.pyd is linked to the
> openmp
> library. The prange tests fail only sometimes. On my system, the prange
> index is sometimes left at the start (zero) of the range, while the
> tests
> expect the index to be left at the stop of the range. According to the
> Cython prange enhancements webpage "the iterations of the loop body
> can be
> executed in any order" .
> Where
> does that leave the loop index?
>

 I should be the value from the last iteration, but in my experience
 many compilers have buggy OpenMP implementations. I think your
 compiler doesn't correctly support the lastprivate clause in all
 situations. For instance, test_prange fails (which doesn't have any
 break/return/exceptions), which simply has a lastprivate clause and a
 schedule clause set to 'dynamic'. The test without the schedule clause
 works fine (or maybe it's just luck). Or maybe it doesn't support
 multiple lastprivate() clauses? I'm not entirely sure... It also seems
 the thread limit on your system is 1.

 In any case, the generated code for these tests looks correct to me,
 but we've had similar problems before with different compilers...
>>>
>>>
>>> A minimal example that fails for me is:
>>>
>>> def test_parallel():
>>>  cdef int i = 0, s = 0
>>>  with nogil, cython.parallel.parallel():
>>>  for i in prange(10):
>>>  s += i
>>>  return i
>>>
>>> The returned value is often 0, otherwise 9 as expected.
>>>
>>> In the generated C code I see
>>>
>>>  #pragma omp for firstprivate(__pyx_v_i) lastprivate(__pyx_v_i)
>>>
>>> If I change this to
>>>
>>>  #pragma omp parallel for firstprivate(__pyx_v_i)
>>> lastprivate(__pyx_v_i)
>>>
>>> the function always returns the expected value. Does that make sense?
>>
>>
>>
>> Well, apparently it doesn't make sense because the value of s is not
>> correct.
>>
>> Christoph
>>
>
> That will use nested parallelism.

(Which means you need 's' to be a reduction)

>>
>>>
>>> Thanks for your help and patience.
>>>
>>> Christoph
>>>
>>>

>
>>
>> The other one is the new "initial_file_path" test that fails with this
>> linker error:
>>
>> """
>> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL
>> /nologo /INCREMENTAL:NO /INCREMENTAL:YES/DEBUG
>> /LIBPATH:X:\Python27\libs
>> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__
>> build\temp.win32-2.7\Release\my_test_package\__init__.obj
>>
>>
>> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd
>>
>> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib
>>
>>
>> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest
>>
>>
>> LINK : error LNK2001: unresolved external symbol init__init__
>> """
>>
>> Maybe the Windows build of distutils is broken here - it seems to
>> assume
>> the wrong module name for the package module.
>
>
>
> I think this is an issue with the test. The extension does compile
> and link
> outside of the tests.
>
>
>
>>
>> I guess compiling package modules is just an overall badly supported
>> feature in CPython...
>>
>>
>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes
>>> during
>>> `test_slice_assignment (memslice.__test__)`. I tested two
>>> computers. The
>>> Windows executive can not identify in which specific module it
>>> crashes,
>>> and
>>> neither enabling faulthandler nor building with debug symbols gives
>>> any
>>> useful information. Can anyone reproduce this? It seems compiler
>>> specific
>>> since Python 3.3, which is using msvc10, does not crash.
>>
>>
>>
>> Hmm, yes, sounds like a problem with the compiler. Would be good to get
>> this sorted out, but it's almost impossible to debug something like
>> this
>> from a distance.
>
>
>
>
> Maybe the following simple example is related. It

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[mark florisson]

On 26 August 2012 22:25, Christoph Gohlke  wrote:
> On 8/26/2012 2:09 PM, Christoph Gohlke wrote:
>>
>> On 8/26/2012 4:08 AM, mark florisson wrote:
>>>
>>> On 25 August 2012 03:07, Christoph Gohlke  wrote:

 Hi,


 On 8/24/2012 12:43 PM, Stefan Behnel wrote:
>
>
> Hi,
>
> thanks for testing!
>
> Christoph Gohlke, 24.08.2012 07:20:
>>
>>
>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers.
>>
>> 32 bit Python 2.7 works well, only 4 test failures.
>
>
>
> Three of those errors are in OpenMP tests - is OpenMP supported in your
> build environment?



 OpenMP is available on my system, and parallel.pyd is linked to the
 openmp
 library. The prange tests fail only sometimes. On my system, the prange
 index is sometimes left at the start (zero) of the range, while the
 tests
 expect the index to be left at the stop of the range. According to the
 Cython prange enhancements webpage "the iterations of the loop body
 can be
 executed in any order" .
 Where
 does that leave the loop index?

>>>
>>> I should be the value from the last iteration, but in my experience
>>> many compilers have buggy OpenMP implementations. I think your
>>> compiler doesn't correctly support the lastprivate clause in all
>>> situations. For instance, test_prange fails (which doesn't have any
>>> break/return/exceptions), which simply has a lastprivate clause and a
>>> schedule clause set to 'dynamic'. The test without the schedule clause
>>> works fine (or maybe it's just luck). Or maybe it doesn't support
>>> multiple lastprivate() clauses? I'm not entirely sure... It also seems
>>> the thread limit on your system is 1.
>>>
>>> In any case, the generated code for these tests looks correct to me,
>>> but we've had similar problems before with different compilers...
>>
>>
>> A minimal example that fails for me is:
>>
>> def test_parallel():
>>  cdef int i = 0, s = 0
>>  with nogil, cython.parallel.parallel():
>>  for i in prange(10):
>>  s += i
>>  return i
>>
>> The returned value is often 0, otherwise 9 as expected.
>>
>> In the generated C code I see
>>
>>  #pragma omp for firstprivate(__pyx_v_i) lastprivate(__pyx_v_i)
>>
>> If I change this to
>>
>>  #pragma omp parallel for firstprivate(__pyx_v_i)
>> lastprivate(__pyx_v_i)
>>
>> the function always returns the expected value. Does that make sense?
>
>
>
> Well, apparently it doesn't make sense because the value of s is not
> correct.
>
> Christoph
>

That will use nested parallelism.

>
>>
>> Thanks for your help and patience.
>>
>> Christoph
>>
>>
>>>

>
> The other one is the new "initial_file_path" test that fails with this
> linker error:
>
> """
> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL
> /nologo /INCREMENTAL:NO /INCREMENTAL:YES/DEBUG
> /LIBPATH:X:\Python27\libs
> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__
> build\temp.win32-2.7\Release\my_test_package\__init__.obj
>
>
> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd
>
> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib
>
>
> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest
>
>
> LINK : error LNK2001: unresolved external symbol init__init__
> """
>
> Maybe the Windows build of distutils is broken here - it seems to
> assume
> the wrong module name for the package module.



 I think this is an issue with the test. The extension does compile
 and link
 outside of the tests.



>
> I guess compiling package modules is just an overall badly supported
> feature in CPython...
>
>
>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes
>> during
>> `test_slice_assignment (memslice.__test__)`. I tested two
>> computers. The
>> Windows executive can not identify in which specific module it
>> crashes,
>> and
>> neither enabling faulthandler nor building with debug symbols gives
>> any
>> useful information. Can anyone reproduce this? It seems compiler
>> specific
>> since Python 3.3, which is using msvc10, does not crash.
>
>
>
> Hmm, yes, sounds like a problem with the compiler. Would be good to get
> this sorted out, but it's almost impossible to debug something like
> this
> from a distance.




 Maybe the following simple example is related. It fails (not crash) when
 compiled with 64 bit msvc9, but does work with 32 bit msvc9 and
 msvc10 (32
 and 64 bit):

 ```
 from cython.view cimport array as cvarray
 import numpy as np

 narr = np.arange(8 * 14 * 11).reshap

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[Christoph Gohlke]

On 8/26/2012 4:08 AM, mark florisson wrote:

On 25 August 2012 03:07, Christoph Gohlke  wrote:

Hi,


On 8/24/2012 12:43 PM, Stefan Behnel wrote:


Hi,

thanks for testing!

Christoph Gohlke, 24.08.2012 07:20:


I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers.

32 bit Python 2.7 works well, only 4 test failures.



Three of those errors are in OpenMP tests - is OpenMP supported in your
build environment?



OpenMP is available on my system, and parallel.pyd is linked to the openmp
library. The prange tests fail only sometimes. On my system, the prange
index is sometimes left at the start (zero) of the range, while the tests
expect the index to be left at the stop of the range. According to the
Cython prange enhancements webpage "the iterations of the loop body can be
executed in any order" . Where
does that leave the loop index?



I should be the value from the last iteration, but in my experience
many compilers have buggy OpenMP implementations. I think your
compiler doesn't correctly support the lastprivate clause in all
situations. For instance, test_prange fails (which doesn't have any
break/return/exceptions), which simply has a lastprivate clause and a
schedule clause set to 'dynamic'. The test without the schedule clause
works fine (or maybe it's just luck). Or maybe it doesn't support
multiple lastprivate() clauses? I'm not entirely sure... It also seems
the thread limit on your system is 1.

In any case, the generated code for these tests looks correct to me,
but we've had similar problems before with different compilers...



On my system, the following patch fixes all the lastprivate related test 
errors and does not have any side effects on other tests. It removes the 
additional initialization of the target index after `#pragma omp parallel`:


diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py
index 188de3d..5ba0eee 100644
--- a/Cython/Compiler/Nodes.py
+++ b/Cython/Compiler/Nodes.py
@@ -7627,7 +7627,7 @@ class ParallelRangeNode(ParallelStatNode):
 # target index uninitialized
 code.putln("if (%(nsteps)s > 0)" % fmt_dict)
 code.begin_block() # if block
-code.putln("%(target)s = 0;" % fmt_dict)
 self.generate_loop(code, fmt_dict)
 code.end_block() # end if block

I'm down to one remaining test failure on win32-py2.7

Christoph









The other one is the new "initial_file_path" test that fails with this
linker error:

"""
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL
/nologo /INCREMENTAL:NO /INCREMENTAL:YES/DEBUG
/LIBPATH:X:\Python27\libs
/LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__
build\temp.win32-2.7\Release\my_test_package\__init__.obj

/OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd
/IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib

/MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest

LINK : error LNK2001: unresolved external symbol init__init__
"""

Maybe the Windows build of distutils is broken here - it seems to assume
the wrong module name for the package module.



I think this is an issue with the test. The extension does compile and link
outside of the tests.





I guess compiling package modules is just an overall badly supported
feature in CPython...



On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes
during
`test_slice_assignment (memslice.__test__)`. I tested two computers. The
Windows executive can not identify in which specific module it crashes,
and
neither enabling faulthandler nor building with debug symbols gives any
useful information. Can anyone reproduce this? It seems compiler specific
since Python 3.3, which is using msvc10, does not crash.



Hmm, yes, sounds like a problem with the compiler. Would be good to get
this sorted out, but it's almost impossible to debug something like this
from a distance.




Maybe the following simple example is related. It fails (not crash) when
compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32
and 64 bit):

```
from cython.view cimport array as cvarray
import numpy as np

narr = np.arange(8 * 14 * 11).reshape((8, 14, 11))

cdef int[:, :, ::1] a = narr
cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3]

print narr[2:8:2, -4:1:-1, 1:3].shape
print b.shape[0], b.shape[1], b.shape[2]
```

On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2)







When disabling
test_slice_assignment, runtests.py completes with many failures.

The results of `runtests.py -v -v` are at
.



The 64bit output looks so broken that I wonder what went wrong here. I
mean, most of the problems look like this:

"""
Expected:
  Traceback (most recent call last):
  TypeError: m() takes at most 2 positional arguments (3 given)
Got:
  Traceback (most recent call last):
  TypeError: m() takes at most 

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[Yaroslav Halchenko]

FWIW -- this beta builds/tests ok across nearly all Debian ports but 4
where process unexpectedly terminates for no obvious reason (it
built/tested just fine on my local sparc box):
https://buildd.debian.org/status/package.php?p=cython&suite=experimental
some time I will check WTF (memory? etc) on those intriguing
ports.

On Thu, 23 Aug 2012, Stefan Behnel wrote:

> Hello everyone,

> on behalf of the Cython project team, I'm proud to announce the release of
> our third beta of Cython 0.17. This is our first and hopefully last release
> candidate for 0.17, another major step forward in the development of the
> language that will make life easier for a lot of users, rounds up some
> rough edges of the compiler and adds (preliminary) support for CPython 3.3
> and PyPy.

> Download: http://cython.org/release/Cython-0.17b3.tar.gz

> Release notes: http://wiki.cython.org/ReleaseNotes-0.17

> Documentation: http://docs.cython.org/

-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834   Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cython 0.17 beta 3 released - release candidate

[Stefan Behnel]

Yaroslav Halchenko, 27.08.2012 03:38:
> FWIW -- this beta builds/tests ok across nearly all Debian ports but 4

Nice! Thanks for testing.


> where process unexpectedly terminates for no obvious reason (it
> built/tested just fine on my local sparc box):
> https://buildd.debian.org/status/package.php?p=cython&suite=experimental
> some time I will check WTF (memory? etc) on those intriguing
> ports.

Could you just give it a rerun? Maybe something or someone external just
terminated them accidentally or by purpose or whatever.

Stefan

___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel