Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
On Fri, Jul 6, 2012 at 2:33 AM, mark florisson
wrote:
> On 6 July 2012 03:34, Dag Sverre Seljebotn wrote:
>>
>>
>> mark florisson wrote:
>>
>>>On 5 July 2012 21:46, Dag Sverre Seljebotn
>>>wrote:
mark florisson wrote:
>On 3 July 2012 20:15, Robert Bradshaw wrote:
>> On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
>> wrote:
>>> On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
On Tue, Jul 3, 2012 at 11:11 AM, Stefan
>>>Behnel
wrote:
>
> Robert Bradshaw, 03.07.2012 19:58:
>>
>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>>
>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>
> I don't know what happens if a C++ exception is not being
>caught, but
> I
> guess it would simply crash the application. That's a bit
>>>more
> visible than
Yep.
> just printing a warning when a Python exception is being
>ignored due
> to a
> missing declaration. It's really unfortunate that our
>documentation
> didn't
> even mention the need for this, because it's not immediately
>obvious
> that
> Cython won't handle errors in "new", and testing for memory
>errors
> isn't
> quite what people commonly do in their test suites.
>
> Apart from that, I agree, users have to take care to
>>>properly
>declare
> the
> API they are using.
Is there any time you do NOT want a "catch (...) {}" block? I
>can't
see a
C++ exception propagating to Python-land doing anything
>>>useful
>ever.
>>>
>>>
>>> That would have been my intuition, too.
>>
>>
>> If it's actually embedded, with the main driver in C++, one
>>>might
>want
>> it to propagate up.
>
>
> But what kind of a propagation would that be? On the way out, it
>could
> induce anything, from side effects to resource leaks to crashes,
> depending
> on what the state of the surrounding code is. It would leave the
>whole
> system in an unpredictable state. I cannot imagine anyone really
>wanting
> this.
>
>
So shouldn't we just make --cplus turn *all* external
>>>functions
>and
methods
(whether C-like or C++-like) into "except +"? (Or keep
>>>except+
>for
manual
translation, but always have a catch(...)".
Performance overhead is the only reason I can think of to not
>do this,
although IIRC C++ catch blocks are only dealt with during
>>>stack
unwinds and
doesn't cost anything/much (?) when they're not triggered.
"except -1" should then actually mean both; "except + except
>-1". So
it's
more a question of just adding catch(...) *everywhere*, than
>making
"except
+" the default.
>>>
>>>
>>> I have no idea if there is a performance impact, but if there
>isn't,
>>> always
>>> catching all exceptions sounds like a reasonable thing to do.
>After
>>> all, we
>>> have no support for catching C++ exceptions on user side.
>>
>>
>> This is a bit like following every C call with "except *"
>>>(though
>the
>> performance ratios are unclear). It just seems a lot to wrap
>every
>> single line of a non-trivial C++ using function with try..catch
>> blocks.
>>>
>>>
>>> It seems "a lot" of just what exactly? Generated code? Binary
>>>size?
>Time
>>> spent in GCC parser?
>>
>> All of the above. And we should take a look at the runtime overhead
>> (which is hopefully nil, but who knows.)
>>
>>> Though I guess one might want to try to pull out the try-catch to
>>>at
>least
>>> only one per code line rather than one per SimpleCallNode.
>>
>> Or even higher, if possible. It's still a lot.
>
>Why would you have to do that? Can't you just insert a try/catch per
>try/except or try/finally block, or if absent, the function body.
>>>That
>will still work with the way temporaries are cleaned up. (It should
>also be implemented for parallel/prange sections).
One disadvantage is that you don't get source code line for the .pyx
>>>file in the stack trace. Which is often exactly the information you are
>>>looking for (even worse, since C++ stack isn't in the stack trace, the
>>>lineno for what seems like the
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
On 8 July 2012 08:59, Robert Bradshaw wrote:
> On Fri, Jul 6, 2012 at 2:33 AM, mark florisson
> wrote:
>> On 6 July 2012 03:34, Dag Sverre Seljebotn
>> wrote:
>>>
>>>
>>> mark florisson wrote:
>>>
On 5 July 2012 21:46, Dag Sverre Seljebotn
wrote:
>
>
> mark florisson wrote:
>
>>On 3 July 2012 20:15, Robert Bradshaw wrote:
>>> On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
>>> wrote:
On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
>
> On Tue, Jul 3, 2012 at 11:11 AM, Stefan
Behnel
> wrote:
>>
>> Robert Bradshaw, 03.07.2012 19:58:
>>>
>>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
Dag Sverre Seljebotn, 03.07.2012 18:11:
>
> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>
>> I don't know what happens if a C++ exception is not being
>>caught, but
>> I
>> guess it would simply crash the application. That's a bit
more
>> visible than
>
>
> Yep.
>
>> just printing a warning when a Python exception is being
>>ignored due
>> to a
>> missing declaration. It's really unfortunate that our
>>documentation
>> didn't
>> even mention the need for this, because it's not immediately
>>obvious
>> that
>> Cython won't handle errors in "new", and testing for memory
>>errors
>> isn't
>> quite what people commonly do in their test suites.
>>
>> Apart from that, I agree, users have to take care to
properly
>>declare
>> the
>> API they are using.
>
>
> Is there any time you do NOT want a "catch (...) {}" block? I
>>can't
> see a
> C++ exception propagating to Python-land doing anything
useful
>>ever.
That would have been my intuition, too.
>>>
>>>
>>> If it's actually embedded, with the main driver in C++, one
might
>>want
>>> it to propagate up.
>>
>>
>> But what kind of a propagation would that be? On the way out, it
>>could
>> induce anything, from side effects to resource leaks to crashes,
>> depending
>> on what the state of the surrounding code is. It would leave the
>>whole
>> system in an unpredictable state. I cannot imagine anyone really
>>wanting
>> this.
>>
>>
> So shouldn't we just make --cplus turn *all* external
functions
>>and
> methods
> (whether C-like or C++-like) into "except +"? (Or keep
except+
>>for
> manual
> translation, but always have a catch(...)".
>
> Performance overhead is the only reason I can think of to not
>>do this,
> although IIRC C++ catch blocks are only dealt with during
stack
> unwinds and
> doesn't cost anything/much (?) when they're not triggered.
>
> "except -1" should then actually mean both; "except + except
>>-1". So
> it's
> more a question of just adding catch(...) *everywhere*, than
>>making
> "except
> +" the default.
I have no idea if there is a performance impact, but if there
>>isn't,
always
catching all exceptions sounds like a reasonable thing to do.
>>After
all, we
have no support for catching C++ exceptions on user side.
>>>
>>>
>>> This is a bit like following every C call with "except *"
(though
>>the
>>> performance ratios are unclear). It just seems a lot to wrap
>>every
>>> single line of a non-trivial C++ using function with try..catch
>>> blocks.
It seems "a lot" of just what exactly? Generated code? Binary
size?
>>Time
spent in GCC parser?
>>>
>>> All of the above. And we should take a look at the runtime overhead
>>> (which is hopefully nil, but who knows.)
>>>
Though I guess one might want to try to pull out the try-catch to
at
>>least
only one per code line rather than one per SimpleCallNode.
>>>
>>> Or even higher, if possible. It's still a lot.
>>
>>Why would you have to do that? Can't you just insert a try/catch per
>>try/except or try/finally block, or if absent, the function body.
That
>>will still work with the way temporaries are cleaned up. (It should
>>also be implemented for parallel/prange sections).
>
> One disadvantage is that you don't get sourc
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
mark florisson, 08.07.2012 11:34: > On 8 July 2012 08:59, Robert Bradshaw wrote: >> On Fri, Jul 6, 2012 at 2:33 AM, mark florisson wrote: >>> On 6 July 2012 03:34, Dag Sverre Seljebotn wrote: I'm not sure if I like C++ exceptions internally in Cython myself. It would mean that C-compiled Cython modules would not be able to call C++-compiled Cython modules, which I think would create a lot of headache. And I think we want to keep the ability to compile with a C compiler if you don't call C++ code... >>> >>> I don't know how often people mix the two. It's probably not worth >>> making a directive either... Are there any platforms where installing >>> a C++ compiler is more of a hassle than a C compiler? E.g. Theano >>> depends on a C++ compiler (and a lot more as well). >> >> Are you implying, for example, that one would have to re-compile NumPy >> with a C++ compiler if you wanted to use any C++ extension modules? >> I'd rather avoid this. > > No, you can still freely mix any extension modules, it's just that > when you cimport stuff from other Cython modules, they have to be ABI > compatible. If C++ is part of the ABI due to how exceptions are > propagated and handled, then those modules need to be compiled both > with C++ (or neither, to get the other semantics with slower (to be > verified) error checks). -1. I don't want to prevent users of the public C-API of lxml from talking to C++ code in their own modules. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] errors in C++ tests
Hi,
I'm seeing C++ compiler errors errors in the C++ tests.
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=cpp,PYVERSION=py3k/491/testReport/
"""
cpp_operators.cpp: In function ‘void __Pyx_CppExn2PyErr()’:
cpp_operators.cpp:442: error: expected unqualified-id before ‘&’ token
cpp_operators.cpp:442: error: expected `)' before ‘&’ token
"""
The failing code line is this:
"""
} catch (const std::bad_alloc& exn) {
"""
That line looks ok to me. It started failing after I enabled C++ error
handling for default constructors, which then lead to the above function
being generated into the module code.
Does anyone know what to make of this?
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
On 8 July 2012 11:17, Stefan Behnel wrote: > mark florisson, 08.07.2012 11:34: >> On 8 July 2012 08:59, Robert Bradshaw wrote: >>> On Fri, Jul 6, 2012 at 2:33 AM, mark florisson wrote: On 6 July 2012 03:34, Dag Sverre Seljebotn wrote: > I'm not sure if I like C++ exceptions internally in Cython myself. > It would mean that C-compiled Cython modules would not be able to call > C++-compiled Cython modules, which I think would create a lot of headache. > And I think we want to keep the ability to compile with a C compiler if > you > don't call C++ code... I don't know how often people mix the two. It's probably not worth making a directive either... Are there any platforms where installing a C++ compiler is more of a hassle than a C compiler? E.g. Theano depends on a C++ compiler (and a lot more as well). >>> >>> Are you implying, for example, that one would have to re-compile NumPy >>> with a C++ compiler if you wanted to use any C++ extension modules? >>> I'd rather avoid this. >> >> No, you can still freely mix any extension modules, it's just that >> when you cimport stuff from other Cython modules, they have to be ABI >> compatible. If C++ is part of the ABI due to how exceptions are >> propagated and handled, then those modules need to be compiled both >> with C++ (or neither, to get the other semantics with slower (to be >> verified) error checks). > > -1. I don't want to prevent users of the public C-API of lxml from talking > to C++ code in their own modules. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel I think you misunderstand. This would only be part of the internal Cython ABI. If you use public/api functions, or take the pointer to the function, you get the C semantics (possibly with unraisable exceptions). The details are in the CEP, this would only replace the extra 'error' argument to cdef functions. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] errors in C++ tests
2012/7/8 Stefan Behnel :
> """
> cpp_operators.cpp: In function ‘void __Pyx_CppExn2PyErr()’:
> cpp_operators.cpp:442: error: expected unqualified-id before ‘&’ token
> cpp_operators.cpp:442: error: expected `)' before ‘&’ token
> """
>
> The failing code line is this:
>
> """
> } catch (const std::bad_alloc& exn) {
> """
Could you check whether the header is included? It should be to
get the definition of bad_alloc.
--
Lars Buitinck
Scientific programmer, ILPS
University of Amsterdam
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] errors in C++ tests
Lars Buitinck, 08.07.2012 20:38:
> 2012/7/8 Stefan Behnel:
>> """
>> cpp_operators.cpp: In function ‘void __Pyx_CppExn2PyErr()’:
>> cpp_operators.cpp:442: error: expected unqualified-id before ‘&’ token
>> cpp_operators.cpp:442: error: expected `)' before ‘&’ token
>> """
>>
>> The failing code line is this:
>>
>> """
>> } catch (const std::bad_alloc& exn) {
>> """
>
> Could you check whether the header is included? It should be to
> get the definition of bad_alloc.
Ah, yes. I'm sure that's it. Are there any side-effects in that header
file, or would it be ok to always include it in C++ mode when the above
exception conversion function is used?
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] errors in C++ tests
Stefan Behnel, 08.07.2012 20:54:
> Lars Buitinck, 08.07.2012 20:38:
>> 2012/7/8 Stefan Behnel:
>>> """
>>> cpp_operators.cpp: In function ‘void __Pyx_CppExn2PyErr()’:
>>> cpp_operators.cpp:442: error: expected unqualified-id before ‘&’ token
>>> cpp_operators.cpp:442: error: expected `)' before ‘&’ token
>>> """
>>>
>>> The failing code line is this:
>>>
>>> """
>>> } catch (const std::bad_alloc& exn) {
>>> """
>>
>> Could you check whether the header is included? It should be to
>> get the definition of bad_alloc.
>
> Ah, yes. I'm sure that's it.
... and there also were some more headers missing, so basically, this
feature never worked. Great. Here's the complete implementation with the
four header files that I had to add at the top:
"""
#include
#include
#include
#include
static void __Pyx_CppExn2PyErr() {
// Catch a handful of different errors here and turn them into the
// equivalent Python errors.
try {
if (PyErr_Occurred())
; // let the latest Python exn pass through and ignore the current one
else
throw;
} catch (const std::bad_alloc& exn) {
PyErr_SetString(PyExc_MemoryError, exn.what());
} catch (const std::bad_cast& exn) {
PyErr_SetString(PyExc_TypeError, exn.what());
} catch (const std::domain_error& exn) {
PyErr_SetString(PyExc_ValueError, exn.what());
} catch (const std::invalid_argument& exn) {
PyErr_SetString(PyExc_ValueError, exn.what());
} catch (const std::ios_base::failure& exn) {
PyErr_SetString(PyExc_IOError, exn.what());
} catch (const std::out_of_range& exn) {
// Change out_of_range to IndexError
PyErr_SetString(PyExc_IndexError, exn.what());
} catch (const std::overflow_error& exn) {
PyErr_SetString(PyExc_OverflowError, exn.what());
} catch (const std::range_error& exn) {
PyErr_SetString(PyExc_ArithmeticError, exn.what());
} catch (const std::underflow_error& exn) {
PyErr_SetString(PyExc_ArithmeticError, exn.what());
} catch (const std::exception& exn) {
PyErr_SetString(PyExc_RuntimeError, exn.what());
}
catch (...)
{
PyErr_SetString(PyExc_RuntimeError, "Unknown exception");
}
}
"""
Back to this question then:
> Are there any side-effects in that header
> file, or would it be ok to always include it in C++ mode when the above
> exception conversion function is used?
Does it make sense to always include these four headers, or should we try
to handle the exceptions conditionally?
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] errors in C++ tests
2012/7/8 Stefan Behnel : > ... and there also were some more headers missing, so basically, this > feature never worked. Great. Here's the complete implementation with the > four header files that I had to add at the top: I'm pretty sure that at some point, it worked. I implemented this in 6291a2 and the five prior commits, but I did add the appropriate headers to Cython/Compiler/Nodes.py. There is a unit test for this feature in tests/run/cpp_exceptions.pyx; that file includes a header that in turn includes , and , but not because it doesn't need that to raise the exception. (It would be more reliable if the test where refactored into C++ modules so that the headers need not be indirectly included.) > #include Actually, you'll want to include where std::ios_base is declared. is a much "heavier" header. > Does it make sense to always include these four headers, or should we try > to handle the exceptions conditionally? As I said, this should already have been done by Cython/Compiler/Nodes.py. I'm not sure what's going wrong. -- Lars Buitinck Scientific programmer, ILPS University of Amsterdam ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] errors in C++ tests
Lars Buitinck, 08.07.2012 22:24: > 2012/7/8 Stefan Behnel: >> ... and there also were some more headers missing, so basically, this >> feature never worked. Great. Here's the complete implementation with the >> four header files that I had to add at the top: > > I'm pretty sure that at some point, it worked. I implemented this in > 6291a2 and the five prior commits, but I did add the appropriate > headers to Cython/Compiler/Nodes.py. > > There is a unit test for this feature in tests/run/cpp_exceptions.pyx; > that file includes a header that in turn includes , and > , but not because it doesn't need that to raise > the exception. (It would be more reliable if the test where refactored > into C++ modules so that the headers need not be indirectly included.) > >> #include > > Actually, you'll want to include where std::ios_base is > declared. is a much "heavier" header. > >> Does it make sense to always include these four headers, or should we try >> to handle the exceptions conditionally? > > As I said, this should already have been done by > Cython/Compiler/Nodes.py. I'm not sure what's going wrong. Ah, I see it now. Those only work for user declared functions. This problem arises for the default construction, i.e. the undeclared no-args constructor that Cython creates automatically. We didn't previously propagate exceptions for it, so the feature did work and I only broke it by fixing the default constructor. Thanks for clearing this up. Any reason why the env.add_include_file() calls would be better than the header inclusion that I implemented right before the function? It's inside of the #ifndef block now. After all, they are only needed by that function, and when users override it by redefining the name from the outside, they do not need to get included. If users include them themselves, they'll be included twice - so what. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] errors in C++ tests
2012/7/8 Stefan Behnel : > Any reason why the env.add_include_file() calls would be better than the > header inclusion that I implemented right before the function? It's inside > of the #ifndef block now. After all, they are only needed by that function, > and when users override it by redefining the name from the outside, they do > not need to get included. If users include them themselves, they'll be > included twice - so what. I'm not really familiar with Cython internals; I just extended Compiler/Nodes.py that previously added only . I observed that this leads to the headers being included once, at the top of the file, which seemed like the clean solution. Multiple inclusion of C++ standard headers should be completely safe, but I don't know whether the C++ standard says anything about including standard headers in the middle of a module. -- Lars Buitinck Scientific programmer, ILPS University of Amsterdam ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
