Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
Robert Bradshaw, 29.06.2012 11:08: > On Thu, Jun 28, 2012 at 10:45 PM, Stefan Behnel wrote: >> Robert Bradshaw, 28.06.2012 21:46: >>> On Thu, Jun 28, 2012 at 11:38 AM, Stefan Behnel wrote: currently, when I write "new CppClass()" in Cython, it generates a straight call to the "new" operator. It doesn't do any error handling. And the current documentation doesn't even mention this case. Is there a "standard" way to handle this? It seems that C++ has different ways to deal with failures here but raises an exception by default. Would you declare the constructor(s) with an "except +MemoryError"? Is there a reason Cython shouldn't be doing this automatically (if nothing else was declared) ? >>> >>> I think it certainly makes sense to declare the default constructor as >>> "except +" (and std::bad_alloc should become MemoryError), >> >> Right. The code in the constructor can raise other exceptions that must >> also be handled properly. An explicit "except +" will handle that. >> >> >>> but whether >>> to implicitly annotate declared constructors is less clear, especially >>> as there's no way to un-annotate them. >> >> I agree, but sadly, it's the default behaviour that is wrong. I'm sure we >> made lots of users run into this trap already. I fixed the documentation >> for now, but the bottom line is that we require users to take care of >> proper declarations themselves. Otherwise, the code that we generate is >> incorrect, although it's 100% certain that an allocation error can occur, >> even if the constructor code doesn't raise any exceptions itself. > > This is always the case. Sure, we always rely on correct declarations. However, this is a case where we *know* that bad things can happen, even if there is no declaration for them. It would be nice to play safe by default. IMHO, only the second best solution would be to raise a warning when we encounter a "new" without an exception declared, so that users are urged into writing safe code. >> Apparently, changing the behaviour of the "new" operator requires a special >> annotation "std::nothrow", which then returns NULL on allocation failures. >> You can pass that from Cython by hacking up a cname, e.g. >> >>Rectangle "(std::nothrow) Rectangle" (int w, int h) >> >> I'm sure there are users out there who figured this out (I mean, I did...) >> and use it in their code, so I agree that this isn't easy to handle because >> Cython simply wouldn't know what the actual error behaviour is for a given >> constructor and how to correctly detect an error. >> >> This problem applies only to heap allocation in that form. However, stack >> allocation and the new exttype field allocation suffer from similar >> problems when the default constructor raises an exception. Exttype fields >> are a particularly nasty case because the user has no control over the >> allocation. A C++ exception in the C++ class constructor would terminate >> the exttype constructor unexpectedly and thus leak resources (in the best >> case - no idea how CPython reacts if you throw a C++ exception through its >> type instantiation code). > > If the default constructor raises an exception then it should be > declared (to not do so is an error on the users part). New raising > bad_alloc is a bit of a special case, but doesn't appl to the stack or > exttype allocations. Right. In those two cases, it would definitely be the fault of the user. >> Similarly, a C++ exception in the constructor of a stack allocated object >> would then originate from the function entry code and potentially hit the >> Python function wrapper etc. Again, potentially leaking resources or worse. >> >> To me, this sounds like we should do something about it. At least for the >> implicit calls to the default constructor, we should generate "except +" >> code automatically because there is no other way to handle them safely. > > If no constructor is declared, it should be "except +" just to be > safe Ok, I'll fix it. > but otherwise I don't see how this is any different than > forgetting to declare exceptions on any other function. Unfortunately > catching exceptions (with custom per-object handling) on a set of > stack allocated objects seems difficult if not impossible (without > resorting to ugly hacks like using placement new everywhere). I don't know what happens if a C++ exception is not being caught, but I guess it would simply crash the application. That's a bit more visible than just printing a warning when a Python exception is being ignored due to a missing declaration. It's really unfortunate that our documentation didn't even mention the need for this, because it's not immediately obvious that Cython won't handle errors in "new", and testing for memory errors isn't quite what people commonly do in their test suites. Apart from that, I agree, users have to take care to properly declare the API they are using. Stefan
Re: [Cython] Cython methods for C/C++ types
On 3 July 2012 05:58, Stefan Behnel wrote: > Hi, > > the discussion on allowing for automatic user provided ways to coerce > between Python types and C++ types got me thinking that this might hide a > rather interesting general feature: methods for low-level types. I faintly > remember that this idea has come up in our discussions before, but here's a > draft CEP for it: > > http://wiki.cython.org/enhancements/ctypemethods > > Basically, it would allow writing this in .pxd files: > > """ > cdef extern from "...": > cdef cppclass MyClass: > cdef int cython_method_here(self): > return 1 > > ctypedef double mydouble: > cdef double cython_method_here(self): > return self ** 2 > > cdef struct mystruct: > int x, y > > cdef int cython_method_here(self): > return self.x + self.y > > cdef union myunion: > int a > double b > > cdef int cython_method_here(self): > return self.a if ImSureImAnInt else self.b > """ > > The C code for these methods would then be generated into the modules that > use these types, similar to what we allow with "__getbuffer__()". Calls > would be direct function calls as with "final" methods. > > I think it fits into what's there in a very natural way. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel Looks a bit like C++. I have no strong feelings either way, it would simplify the memoryview implementation slightly, which special-cases the memoryview struct with method attributes. I would prefer to keep the language simple however, and have people write functions operating on structs. It's really the same thing written differently, and methods don't really bring an advantage since the data in the struct cannot be hidden or readonly. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython methods for C/C++ types
On 3 July 2012 05:58, Stefan Behnel wrote: > Hi, > > the discussion on allowing for automatic user provided ways to coerce > between Python types and C++ types got me thinking that this might hide a > rather interesting general feature: methods for low-level types. I faintly > remember that this idea has come up in our discussions before, but here's a > draft CEP for it: > > http://wiki.cython.org/enhancements/ctypemethods I see this CEP also mentions overriding C++ methods. But I think if it doesn't support virtual methods, then overriding base class methods by calling another function is pretty horrible. > Basically, it would allow writing this in .pxd files: > > """ > cdef extern from "...": > cdef cppclass MyClass: > cdef int cython_method_here(self): > return 1 > > ctypedef double mydouble: > cdef double cython_method_here(self): > return self ** 2 > > cdef struct mystruct: > int x, y > > cdef int cython_method_here(self): > return self.x + self.y > > cdef union myunion: > int a > double b > > cdef int cython_method_here(self): > return self.a if ImSureImAnInt else self.b > """ > > The C code for these methods would then be generated into the modules that > use these types, similar to what we allow with "__getbuffer__()". Calls > would be direct function calls as with "final" methods. > > I think it fits into what's there in a very natural way. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Exception: Tests in module 'parallel' were unexpectedly killed by signal 11
On 3 July 2012 00:04, Yaroslav Halchenko wrote: > please pardon my sloppy cut/paste which ruined the subject line in the > original post ;) So the topic is: > > On Mon, 02 Jul 2012, Yaroslav Halchenko wrote: >> I was thinking about updating Debian package for cython 0.16 but ran into >> the failing unittests so decided first to check with experts: anything >> obvious >> which comes to mind from seeing those? > >> ... >> compiling (c) and running sequential_parallel ... sequential_parallel.c: In >> function '__pyx_pf_19sequential_parallel_56test_chunksize': >> sequential_parallel.c:12405:7: warning: variable '__pyx_t_6' set but not >> used [-Wunused-but-set-variable] >> sequential_parallel.c:12404:7: warning: variable '__pyx_t_5' set but not >> used [-Wunused-but-set-variable] >> sequential_parallel.c:12402:7: warning: variable '__pyx_t_3' set but not >> used [-Wunused-but-set-variable] >> sequential_parallel.c: In function >> '__pyx_pw_19sequential_parallel_25test_nested_break_continue': >> sequential_parallel.c:5887:29: warning: '__pyx_v_j' may be used >> uninitialized in this function [-Wmaybe-uninitialized] >> sequential_parallel.c:5590:7: note: '__pyx_v_j' was declared here >> sequential_parallel.c: In function >> '__pyx_f_19sequential_parallel_parallel_exc_cpdef_unnested.isra.29': >> sequential_parallel.c:8915:3: warning: '__pyx_r' may be used uninitialized >> in this function [-Wmaybe-uninitialized] >> sequential_parallel.c:8681:7: note: '__pyx_r' was declared here >> sequential_parallel.c: In function >> '__pyx_f_19sequential_parallel_parallel_exc_cpdef.isra.30': >> sequential_parallel.c:8614:3: warning: '__pyx_r' may be used uninitialized >> in this function [-Wmaybe-uninitialized] >> sequential_parallel.c:8259:7: note: '__pyx_r' was declared here >> sequential_parallel.c: In function >> '__pyx_pw_19sequential_parallel_33test_parallel_exc_cdef': >> sequential_parallel.c:8222:78: warning: '__pyx_r' may be used uninitialized >> in this function [-Wuninitialized] >> sequential_parallel.c:7946:7: note: '__pyx_r' was declared here >> sequential_parallel.c:8233:69: warning: '__pyx_r' may be used uninitialized >> in this function [-Wuninitialized] >> sequential_parallel.c:7578:7: note: '__pyx_r' was declared here >> Fatal Python error: deletion of interned string failed >> Compiling >> /tmp/buildd/.cython/inline/_cython_inline_a998a12d3d1450b2368ca75e07a27942.pyx >> because it changed. >> Cythonizing >> /tmp/buildd/.cython/inline/_cython_inline_a998a12d3d1450b2368ca75e07a27942.pyx >> ERROR >> ... >> == >> ERROR: compiling (c) and running parallel >> -- >> Traceback (most recent call last): >> File "runtests.py", line 707, in run >> self.run_tests(result) >> File "runtests.py", line 723, in run_tests >> self.run_doctests(self.module, result) >> File "runtests.py", line 729, in run_doctests >> run_forked_test(result, run_test, self.shortDescription(), self.fork) >> File "runtests.py", line 778, in run_forked_test >> (module_name, result_code & 255)) >> Exception: Tests in module 'parallel' were unexpectedly killed by signal 11 > >> == > -- > Yaroslav O. Halchenko > Postdoctoral Fellow, Department of Psychological and Brain Sciences > Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 > Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 > WWW: http://www.linkedin.com/in/yarik > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel Thanks for the report, I'll have a look. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython methods for C/C++ types
mark florisson, 03.07.2012 12:15: > On 3 July 2012 05:58, Stefan Behnel wrote: >> the discussion on allowing for automatic user provided ways to coerce >> between Python types and C++ types got me thinking that this might hide a >> rather interesting general feature: methods for low-level types. I faintly >> remember that this idea has come up in our discussions before, but here's a >> draft CEP for it: >> >> http://wiki.cython.org/enhancements/ctypemethods > > I see this CEP also mentions overriding C++ methods. But I think if it > doesn't support virtual methods, then overriding base class methods by > calling another function is pretty horrible. Right, I didn't think that part through. If we don't know the real type at compile time and get a subtype at runtime then we'll call the wrong code. I agree that that's not a desirable feature. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
On 07/03/2012 09:14 AM, Stefan Behnel wrote:
Robert Bradshaw, 29.06.2012 11:08:
On Thu, Jun 28, 2012 at 10:45 PM, Stefan Behnel wrote:
Robert Bradshaw, 28.06.2012 21:46:
On Thu, Jun 28, 2012 at 11:38 AM, Stefan Behnel wrote:
currently, when I write "new CppClass()" in Cython, it generates a straight
call to the "new" operator. It doesn't do any error handling. And the
current documentation doesn't even mention this case.
Is there a "standard" way to handle this? It seems that C++ has different
ways to deal with failures here but raises an exception by default. Would
you declare the constructor(s) with an "except +MemoryError"? Is there a
reason Cython shouldn't be doing this automatically (if nothing else was
declared) ?
I think it certainly makes sense to declare the default constructor as
"except +" (and std::bad_alloc should become MemoryError),
Right. The code in the constructor can raise other exceptions that must
also be handled properly. An explicit "except +" will handle that.
but whether
to implicitly annotate declared constructors is less clear, especially
as there's no way to un-annotate them.
I agree, but sadly, it's the default behaviour that is wrong. I'm sure we
made lots of users run into this trap already. I fixed the documentation
for now, but the bottom line is that we require users to take care of
proper declarations themselves. Otherwise, the code that we generate is
incorrect, although it's 100% certain that an allocation error can occur,
even if the constructor code doesn't raise any exceptions itself.
This is always the case.
Sure, we always rely on correct declarations. However, this is a case where
we *know* that bad things can happen, even if there is no declaration for
them. It would be nice to play safe by default.
IMHO, only the second best solution would be to raise a warning when we
encounter a "new" without an exception declared, so that users are urged
into writing safe code.
Apparently, changing the behaviour of the "new" operator requires a special
annotation "std::nothrow", which then returns NULL on allocation failures.
You can pass that from Cython by hacking up a cname, e.g.
Rectangle "(std::nothrow) Rectangle" (int w, int h)
I'm sure there are users out there who figured this out (I mean, I did...)
and use it in their code, so I agree that this isn't easy to handle because
Cython simply wouldn't know what the actual error behaviour is for a given
constructor and how to correctly detect an error.
This problem applies only to heap allocation in that form. However, stack
allocation and the new exttype field allocation suffer from similar
problems when the default constructor raises an exception. Exttype fields
are a particularly nasty case because the user has no control over the
allocation. A C++ exception in the C++ class constructor would terminate
the exttype constructor unexpectedly and thus leak resources (in the best
case - no idea how CPython reacts if you throw a C++ exception through its
type instantiation code).
If the default constructor raises an exception then it should be
declared (to not do so is an error on the users part). New raising
bad_alloc is a bit of a special case, but doesn't appl to the stack or
exttype allocations.
Right. In those two cases, it would definitely be the fault of the user.
Similarly, a C++ exception in the constructor of a stack allocated object
would then originate from the function entry code and potentially hit the
Python function wrapper etc. Again, potentially leaking resources or worse.
To me, this sounds like we should do something about it. At least for the
implicit calls to the default constructor, we should generate "except +"
code automatically because there is no other way to handle them safely.
If no constructor is declared, it should be "except +" just to be
safe
Ok, I'll fix it.
but otherwise I don't see how this is any different than
forgetting to declare exceptions on any other function. Unfortunately
catching exceptions (with custom per-object handling) on a set of
stack allocated objects seems difficult if not impossible (without
resorting to ugly hacks like using placement new everywhere).
I don't know what happens if a C++ exception is not being caught, but I
guess it would simply crash the application. That's a bit more visible than
Yep.
just printing a warning when a Python exception is being ignored due to a
missing declaration. It's really unfortunate that our documentation didn't
even mention the need for this, because it's not immediately obvious that
Cython won't handle errors in "new", and testing for memory errors isn't
quite what people commonly do in their test suites.
Apart from that, I agree, users have to take care to properly declare the
API they are using.
Is there any time you do NOT want a "catch (...) {}" block? I can't see
a C++ exception propagating to Python-land doing anything useful ever.
So shouldn't we just make --cplus tur
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
Dag Sverre Seljebotn, 03.07.2012 18:11:
> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>> I don't know what happens if a C++ exception is not being caught, but I
>> guess it would simply crash the application. That's a bit more visible than
>
> Yep.
>
>> just printing a warning when a Python exception is being ignored due to a
>> missing declaration. It's really unfortunate that our documentation didn't
>> even mention the need for this, because it's not immediately obvious that
>> Cython won't handle errors in "new", and testing for memory errors isn't
>> quite what people commonly do in their test suites.
>>
>> Apart from that, I agree, users have to take care to properly declare the
>> API they are using.
>
> Is there any time you do NOT want a "catch (...) {}" block? I can't see a
> C++ exception propagating to Python-land doing anything useful ever.
That would have been my intuition, too.
> So shouldn't we just make --cplus turn *all* external functions and methods
> (whether C-like or C++-like) into "except +"? (Or keep except+ for manual
> translation, but always have a catch(...)".
>
> Performance overhead is the only reason I can think of to not do this,
> although IIRC C++ catch blocks are only dealt with during stack unwinds and
> doesn't cost anything/much (?) when they're not triggered.
>
> "except -1" should then actually mean both; "except + except -1". So it's
> more a question of just adding catch(...) *everywhere*, than making "except
> +" the default.
I have no idea if there is a performance impact, but if there isn't, always
catching all exceptions sounds like a reasonable thing to do. After all, we
have no support for catching C++ exceptions on user side.
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 03.07.2012 18:11:
>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>> I don't know what happens if a C++ exception is not being caught, but I
>>> guess it would simply crash the application. That's a bit more visible than
>>
>> Yep.
>>
>>> just printing a warning when a Python exception is being ignored due to a
>>> missing declaration. It's really unfortunate that our documentation didn't
>>> even mention the need for this, because it's not immediately obvious that
>>> Cython won't handle errors in "new", and testing for memory errors isn't
>>> quite what people commonly do in their test suites.
>>>
>>> Apart from that, I agree, users have to take care to properly declare the
>>> API they are using.
>>
>> Is there any time you do NOT want a "catch (...) {}" block? I can't see a
>> C++ exception propagating to Python-land doing anything useful ever.
>
> That would have been my intuition, too.
If it's actually embedded, with the main driver in C++, one might want
it to propagate up.
>> So shouldn't we just make --cplus turn *all* external functions and methods
>> (whether C-like or C++-like) into "except +"? (Or keep except+ for manual
>> translation, but always have a catch(...)".
>>
>> Performance overhead is the only reason I can think of to not do this,
>> although IIRC C++ catch blocks are only dealt with during stack unwinds and
>> doesn't cost anything/much (?) when they're not triggered.
>>
>> "except -1" should then actually mean both; "except + except -1". So it's
>> more a question of just adding catch(...) *everywhere*, than making "except
>> +" the default.
>
> I have no idea if there is a performance impact, but if there isn't, always
> catching all exceptions sounds like a reasonable thing to do. After all, we
> have no support for catching C++ exceptions on user side.
This is a bit like following every C call with "except *" (though the
performance ratios are unclear). It just seems a lot to wrap every
single line of a non-trivial C++ using function with try..catch
blocks. I also don't think this would play well with Pynac (from Sage)
which is a C++ library with Cython callbacks that may call back into
the library and raise C++ exceptions (but that does feel a bit risky
anyways).
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
Robert Bradshaw, 03.07.2012 19:58:
> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
I don't know what happens if a C++ exception is not being caught, but I
guess it would simply crash the application. That's a bit more visible than
>>>
>>> Yep.
>>>
just printing a warning when a Python exception is being ignored due to a
missing declaration. It's really unfortunate that our documentation didn't
even mention the need for this, because it's not immediately obvious that
Cython won't handle errors in "new", and testing for memory errors isn't
quite what people commonly do in their test suites.
Apart from that, I agree, users have to take care to properly declare the
API they are using.
>>>
>>> Is there any time you do NOT want a "catch (...) {}" block? I can't see a
>>> C++ exception propagating to Python-land doing anything useful ever.
>>
>> That would have been my intuition, too.
>
> If it's actually embedded, with the main driver in C++, one might want
> it to propagate up.
But what kind of a propagation would that be? On the way out, it could
induce anything, from side effects to resource leaks to crashes, depending
on what the state of the surrounding code is. It would leave the whole
system in an unpredictable state. I cannot imagine anyone really wanting this.
>>> So shouldn't we just make --cplus turn *all* external functions and methods
>>> (whether C-like or C++-like) into "except +"? (Or keep except+ for manual
>>> translation, but always have a catch(...)".
>>>
>>> Performance overhead is the only reason I can think of to not do this,
>>> although IIRC C++ catch blocks are only dealt with during stack unwinds and
>>> doesn't cost anything/much (?) when they're not triggered.
>>>
>>> "except -1" should then actually mean both; "except + except -1". So it's
>>> more a question of just adding catch(...) *everywhere*, than making "except
>>> +" the default.
>>
>> I have no idea if there is a performance impact, but if there isn't, always
>> catching all exceptions sounds like a reasonable thing to do. After all, we
>> have no support for catching C++ exceptions on user side.
>
> This is a bit like following every C call with "except *" (though the
> performance ratios are unclear). It just seems a lot to wrap every
> single line of a non-trivial C++ using function with try..catch
> blocks.
But if users are correct about their declarations, we'd end up with the
same thing. I think it's worth a try.
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel wrote:
> Robert Bradshaw, 03.07.2012 19:58:
>> On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>>> Dag Sverre Seljebotn, 03.07.2012 18:11:
On 07/03/2012 09:14 AM, Stefan Behnel wrote:
> I don't know what happens if a C++ exception is not being caught, but I
> guess it would simply crash the application. That's a bit more visible
> than
Yep.
> just printing a warning when a Python exception is being ignored due to a
> missing declaration. It's really unfortunate that our documentation didn't
> even mention the need for this, because it's not immediately obvious that
> Cython won't handle errors in "new", and testing for memory errors isn't
> quite what people commonly do in their test suites.
>
> Apart from that, I agree, users have to take care to properly declare the
> API they are using.
Is there any time you do NOT want a "catch (...) {}" block? I can't see a
C++ exception propagating to Python-land doing anything useful ever.
>>>
>>> That would have been my intuition, too.
>>
>> If it's actually embedded, with the main driver in C++, one might want
>> it to propagate up.
>
> But what kind of a propagation would that be? On the way out, it could
> induce anything, from side effects to resource leaks to crashes, depending
> on what the state of the surrounding code is. It would leave the whole
> system in an unpredictable state. I cannot imagine anyone really wanting this.
>
>
So shouldn't we just make --cplus turn *all* external functions and methods
(whether C-like or C++-like) into "except +"? (Or keep except+ for manual
translation, but always have a catch(...)".
Performance overhead is the only reason I can think of to not do this,
although IIRC C++ catch blocks are only dealt with during stack unwinds and
doesn't cost anything/much (?) when they're not triggered.
"except -1" should then actually mean both; "except + except -1". So it's
more a question of just adding catch(...) *everywhere*, than making "except
+" the default.
>>>
>>> I have no idea if there is a performance impact, but if there isn't, always
>>> catching all exceptions sounds like a reasonable thing to do. After all, we
>>> have no support for catching C++ exceptions on user side.
>>
>> This is a bit like following every C call with "except *" (though the
>> performance ratios are unclear). It just seems a lot to wrap every
>> single line of a non-trivial C++ using function with try..catch
>> blocks.
>
> But if users are correct about their declarations, we'd end up with the
> same thing. I think it's worth a try.
Most C++ code (that I've ever run into) doesn't use exceptions,
because exception handling is so broken in C++ anyways.
- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel wrote:
Robert Bradshaw, 03.07.2012 19:58:
On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
Dag Sverre Seljebotn, 03.07.2012 18:11:
On 07/03/2012 09:14 AM, Stefan Behnel wrote:
I don't know what happens if a C++ exception is not being caught, but I
guess it would simply crash the application. That's a bit more visible than
Yep.
just printing a warning when a Python exception is being ignored due to a
missing declaration. It's really unfortunate that our documentation didn't
even mention the need for this, because it's not immediately obvious that
Cython won't handle errors in "new", and testing for memory errors isn't
quite what people commonly do in their test suites.
Apart from that, I agree, users have to take care to properly declare the
API they are using.
Is there any time you do NOT want a "catch (...) {}" block? I can't see a
C++ exception propagating to Python-land doing anything useful ever.
That would have been my intuition, too.
If it's actually embedded, with the main driver in C++, one might want
it to propagate up.
But what kind of a propagation would that be? On the way out, it could
induce anything, from side effects to resource leaks to crashes, depending
on what the state of the surrounding code is. It would leave the whole
system in an unpredictable state. I cannot imagine anyone really wanting this.
So shouldn't we just make --cplus turn *all* external functions and methods
(whether C-like or C++-like) into "except +"? (Or keep except+ for manual
translation, but always have a catch(...)".
Performance overhead is the only reason I can think of to not do this,
although IIRC C++ catch blocks are only dealt with during stack unwinds and
doesn't cost anything/much (?) when they're not triggered.
"except -1" should then actually mean both; "except + except -1". So it's
more a question of just adding catch(...) *everywhere*, than making "except
+" the default.
I have no idea if there is a performance impact, but if there isn't, always
catching all exceptions sounds like a reasonable thing to do. After all, we
have no support for catching C++ exceptions on user side.
This is a bit like following every C call with "except *" (though the
performance ratios are unclear). It just seems a lot to wrap every
single line of a non-trivial C++ using function with try..catch
blocks.
It seems "a lot" of just what exactly? Generated code? Binary size? Time
spent in GCC parser?
Though I guess one might want to try to pull out the try-catch to at
least only one per code line rather than one per SimpleCallNode.
"except *" only has a point when calling functions using the CPython
API, but most external C functions are pure C, not
CPython-API-using-functions. OTOH, all external C++ functions are C++ :-)
(Also, if we wrote Cython from scratch now I'm pretty sure the "except
*" defaults would be a tad different.)
But if users are correct about their declarations, we'd end up with the
same thing. I think it's worth a try.
Most C++ code (that I've ever run into) doesn't use exceptions,
because exception handling is so broken in C++ anyways.
Except for the fact that any code touching "new" could be raising
exceptions? That propagates.
There is a lot of C++ code out there using exceptions. I'd guess that
both mathematical code and Google-written code is unlike most C++ code
out there :-) Many C++ programmers go on and on about RAII and auto_ptrs
and so on, and that doesn't have much point unless you throw an
exception now and then (OK, there's the occasional return statement
where it matters well).
Dag
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel wrote:
Robert Bradshaw, 03.07.2012 19:58:
On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
Dag Sverre Seljebotn, 03.07.2012 18:11:
On 07/03/2012 09:14 AM, Stefan Behnel wrote:
I don't know what happens if a C++ exception is not being caught, but I
guess it would simply crash the application. That's a bit more visible than
Yep.
just printing a warning when a Python exception is being ignored due to a
missing declaration. It's really unfortunate that our documentation didn't
even mention the need for this, because it's not immediately obvious that
Cython won't handle errors in "new", and testing for memory errors isn't
quite what people commonly do in their test suites.
Apart from that, I agree, users have to take care to properly declare the
API they are using.
Is there any time you do NOT want a "catch (...) {}" block? I can't see a
C++ exception propagating to Python-land doing anything useful ever.
That would have been my intuition, too.
If it's actually embedded, with the main driver in C++, one might want
it to propagate up.
But what kind of a propagation would that be? On the way out, it could
induce anything, from side effects to resource leaks to crashes, depending
on what the state of the surrounding code is. It would leave the whole
system in an unpredictable state. I cannot imagine anyone really wanting this.
So shouldn't we just make --cplus turn *all* external functions and methods
(whether C-like or C++-like) into "except +"? (Or keep except+ for manual
translation, but always have a catch(...)".
Performance overhead is the only reason I can think of to not do this,
although IIRC C++ catch blocks are only dealt with during stack unwinds and
doesn't cost anything/much (?) when they're not triggered.
"except -1" should then actually mean both; "except + except -1". So it's
more a question of just adding catch(...) *everywhere*, than making "except
+" the default.
I have no idea if there is a performance impact, but if there isn't, always
catching all exceptions sounds like a reasonable thing to do. After all, we
have no support for catching C++ exceptions on user side.
This is a bit like following every C call with "except *" (though the
performance ratios are unclear). It just seems a lot to wrap every
single line of a non-trivial C++ using function with try..catch
blocks.
But if users are correct about their declarations, we'd end up with the
same thing. I think it's worth a try.
Most C++ code (that I've ever run into) doesn't use exceptions,
because exception handling is so broken in C++ anyways.
Did you mean that most C++ code doesn't use try/except? That's true, but
my experience is that throw is used.
Dag
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
wrote:
> On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
>>
>> On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel
>> wrote:
>>>
>>> Robert Bradshaw, 03.07.2012 19:58:
On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
>
> Dag Sverre Seljebotn, 03.07.2012 18:11:
>>
>> On 07/03/2012 09:14 AM, Stefan Behnel wrote:
>>>
>>> I don't know what happens if a C++ exception is not being caught, but
>>> I
>>> guess it would simply crash the application. That's a bit more
>>> visible than
>>
>>
>> Yep.
>>
>>> just printing a warning when a Python exception is being ignored due
>>> to a
>>> missing declaration. It's really unfortunate that our documentation
>>> didn't
>>> even mention the need for this, because it's not immediately obvious
>>> that
>>> Cython won't handle errors in "new", and testing for memory errors
>>> isn't
>>> quite what people commonly do in their test suites.
>>>
>>> Apart from that, I agree, users have to take care to properly declare
>>> the
>>> API they are using.
>>
>>
>> Is there any time you do NOT want a "catch (...) {}" block? I can't
>> see a
>> C++ exception propagating to Python-land doing anything useful ever.
>
>
> That would have been my intuition, too.
If it's actually embedded, with the main driver in C++, one might want
it to propagate up.
>>>
>>>
>>> But what kind of a propagation would that be? On the way out, it could
>>> induce anything, from side effects to resource leaks to crashes,
>>> depending
>>> on what the state of the surrounding code is. It would leave the whole
>>> system in an unpredictable state. I cannot imagine anyone really wanting
>>> this.
>>>
>>>
>> So shouldn't we just make --cplus turn *all* external functions and
>> methods
>> (whether C-like or C++-like) into "except +"? (Or keep except+ for
>> manual
>> translation, but always have a catch(...)".
>>
>> Performance overhead is the only reason I can think of to not do this,
>> although IIRC C++ catch blocks are only dealt with during stack
>> unwinds and
>> doesn't cost anything/much (?) when they're not triggered.
>>
>> "except -1" should then actually mean both; "except + except -1". So
>> it's
>> more a question of just adding catch(...) *everywhere*, than making
>> "except
>> +" the default.
>
>
> I have no idea if there is a performance impact, but if there isn't,
> always
> catching all exceptions sounds like a reasonable thing to do. After
> all, we
> have no support for catching C++ exceptions on user side.
This is a bit like following every C call with "except *" (though the
performance ratios are unclear). It just seems a lot to wrap every
single line of a non-trivial C++ using function with try..catch
blocks.
>
>
> It seems "a lot" of just what exactly? Generated code? Binary size? Time
> spent in GCC parser?
All of the above. And we should take a look at the runtime overhead
(which is hopefully nil, but who knows.)
> Though I guess one might want to try to pull out the try-catch to at least
> only one per code line rather than one per SimpleCallNode.
Or even higher, if possible. It's still a lot.
> "except *" only has a point when calling functions using the CPython API,
> but most external C functions are pure C, not CPython-API-using-functions.
> OTOH, all external C++ functions are C++ :-)
Fair point.
> (Also, if we wrote Cython from scratch now I'm pretty sure the "except *"
> defaults would be a tad different.)
For sure.
>>> But if users are correct about their declarations, we'd end up with the
>>> same thing. I think it's worth a try.
>>
>>
>> Most C++ code (that I've ever run into) doesn't use exceptions,
>> because exception handling is so broken in C++ anyways.
>
>
> Except for the fact that any code touching "new" could be raising
> exceptions? That propagates.
I would guess most of the time people don't bother catching these and
let the program die, as there's often no sane recovery (the same as
MemoryErrors in Python, though I guess C++ is less often used from an
event loop).
> There is a lot of C++ code out there using exceptions. I'd guess that both
> mathematical code and Google-written code is unlike most C++ code out there
> :-) Many C++ programmers go on and on about RAII and auto_ptrs and so on,
> and that doesn't have much point unless you throw an exception now and then
> (OK, there's the occasional return statement where it matters well).
True, I've seen a small subset of the C++ code that's out there. Maybe
numerical computations use it a lot?
+1 to making catch-everywhere a directive at least. Lets see what the
impact is before we decide to make it the default.
- Robert
___
cyth
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
On 07/03/2012 09:15 PM, Robert Bradshaw wrote:
On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
wrote:
On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel
wrote:
Robert Bradshaw, 03.07.2012 19:58:
On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
Dag Sverre Seljebotn, 03.07.2012 18:11:
On 07/03/2012 09:14 AM, Stefan Behnel wrote:
I don't know what happens if a C++ exception is not being caught, but
I
guess it would simply crash the application. That's a bit more
visible than
Yep.
just printing a warning when a Python exception is being ignored due
to a
missing declaration. It's really unfortunate that our documentation
didn't
even mention the need for this, because it's not immediately obvious
that
Cython won't handle errors in "new", and testing for memory errors
isn't
quite what people commonly do in their test suites.
Apart from that, I agree, users have to take care to properly declare
the
API they are using.
Is there any time you do NOT want a "catch (...) {}" block? I can't
see a
C++ exception propagating to Python-land doing anything useful ever.
That would have been my intuition, too.
If it's actually embedded, with the main driver in C++, one might want
it to propagate up.
But what kind of a propagation would that be? On the way out, it could
induce anything, from side effects to resource leaks to crashes,
depending
on what the state of the surrounding code is. It would leave the whole
system in an unpredictable state. I cannot imagine anyone really wanting
this.
So shouldn't we just make --cplus turn *all* external functions and
methods
(whether C-like or C++-like) into "except +"? (Or keep except+ for
manual
translation, but always have a catch(...)".
Performance overhead is the only reason I can think of to not do this,
although IIRC C++ catch blocks are only dealt with during stack
unwinds and
doesn't cost anything/much (?) when they're not triggered.
"except -1" should then actually mean both; "except + except -1". So
it's
more a question of just adding catch(...) *everywhere*, than making
"except
+" the default.
I have no idea if there is a performance impact, but if there isn't,
always
catching all exceptions sounds like a reasonable thing to do. After
all, we
have no support for catching C++ exceptions on user side.
This is a bit like following every C call with "except *" (though the
performance ratios are unclear). It just seems a lot to wrap every
single line of a non-trivial C++ using function with try..catch
blocks.
It seems "a lot" of just what exactly? Generated code? Binary size? Time
spent in GCC parser?
All of the above. And we should take a look at the runtime overhead
(which is hopefully nil, but who knows.)
Though I guess one might want to try to pull out the try-catch to at least
only one per code line rather than one per SimpleCallNode.
Or even higher, if possible. It's still a lot.
"except *" only has a point when calling functions using the CPython API,
but most external C functions are pure C, not CPython-API-using-functions.
OTOH, all external C++ functions are C++ :-)
Fair point.
(Also, if we wrote Cython from scratch now I'm pretty sure the "except *"
defaults would be a tad different.)
For sure.
But if users are correct about their declarations, we'd end up with the
same thing. I think it's worth a try.
Most C++ code (that I've ever run into) doesn't use exceptions,
because exception handling is so broken in C++ anyways.
Except for the fact that any code touching "new" could be raising
exceptions? That propagates.
I would guess most of the time people don't bother catching these and
let the program die, as there's often no sane recovery (the same as
MemoryErrors in Python, though I guess C++ is less often used from an
event loop).
The point is to provide the user with the stack trace of exactly where
you ran out of memory. Even if you can't follow it deep into C++-land,
the Python-side stack trace can be a worth a lot.
There is a lot of C++ code out there using exceptions. I'd guess that both
mathematical code and Google-written code is unlike most C++ code out there
:-) Many C++ programmers go on and on about RAII and auto_ptrs and so on,
and that doesn't have much point unless you throw an exception now and then
(OK, there's the occasional return statement where it matters well).
True, I've seen a small subset of the C++ code that's out there. Maybe
numerical computations use it a lot?
One library I'm using for distributed dense linear algebra (Elemental)
does. Numerical computations libraries are very different from one another.
Dag
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
On 07/03/2012 10:39 PM, Dag Sverre Seljebotn wrote:
On 07/03/2012 09:15 PM, Robert Bradshaw wrote:
On Tue, Jul 3, 2012 at 11:43 AM, Dag Sverre Seljebotn
wrote:
On 07/03/2012 08:23 PM, Robert Bradshaw wrote:
On Tue, Jul 3, 2012 at 11:11 AM, Stefan Behnel
wrote:
Robert Bradshaw, 03.07.2012 19:58:
On Tue, Jul 3, 2012 at 9:38 AM, Stefan Behnel wrote:
Dag Sverre Seljebotn, 03.07.2012 18:11:
On 07/03/2012 09:14 AM, Stefan Behnel wrote:
I don't know what happens if a C++ exception is not being
caught, but
I
guess it would simply crash the application. That's a bit more
visible than
Yep.
just printing a warning when a Python exception is being
ignored due
to a
missing declaration. It's really unfortunate that our
documentation
didn't
even mention the need for this, because it's not immediately
obvious
that
Cython won't handle errors in "new", and testing for memory errors
isn't
quite what people commonly do in their test suites.
Apart from that, I agree, users have to take care to properly
declare
the
API they are using.
Is there any time you do NOT want a "catch (...) {}" block? I can't
see a
C++ exception propagating to Python-land doing anything useful
ever.
That would have been my intuition, too.
If it's actually embedded, with the main driver in C++, one might
want
it to propagate up.
But what kind of a propagation would that be? On the way out, it could
induce anything, from side effects to resource leaks to crashes,
depending
on what the state of the surrounding code is. It would leave the whole
system in an unpredictable state. I cannot imagine anyone really
wanting
this.
So shouldn't we just make --cplus turn *all* external functions and
methods
(whether C-like or C++-like) into "except +"? (Or keep except+ for
manual
translation, but always have a catch(...)".
Performance overhead is the only reason I can think of to not do
this,
although IIRC C++ catch blocks are only dealt with during stack
unwinds and
doesn't cost anything/much (?) when they're not triggered.
"except -1" should then actually mean both; "except + except
-1". So
it's
more a question of just adding catch(...) *everywhere*, than making
"except
+" the default.
I have no idea if there is a performance impact, but if there isn't,
always
catching all exceptions sounds like a reasonable thing to do. After
all, we
have no support for catching C++ exceptions on user side.
This is a bit like following every C call with "except *" (though the
performance ratios are unclear). It just seems a lot to wrap every
single line of a non-trivial C++ using function with try..catch
blocks.
It seems "a lot" of just what exactly? Generated code? Binary size? Time
spent in GCC parser?
All of the above. And we should take a look at the runtime overhead
(which is hopefully nil, but who knows.)
Though I guess one might want to try to pull out the try-catch to at
least
only one per code line rather than one per SimpleCallNode.
Or even higher, if possible. It's still a lot.
"except *" only has a point when calling functions using the CPython
API,
but most external C functions are pure C, not
CPython-API-using-functions.
OTOH, all external C++ functions are C++ :-)
Fair point.
(Also, if we wrote Cython from scratch now I'm pretty sure the
"except *"
defaults would be a tad different.)
For sure.
But if users are correct about their declarations, we'd end up with
the
same thing. I think it's worth a try.
Most C++ code (that I've ever run into) doesn't use exceptions,
because exception handling is so broken in C++ anyways.
Except for the fact that any code touching "new" could be raising
exceptions? That propagates.
I would guess most of the time people don't bother catching these and
let the program die, as there's often no sane recovery (the same as
MemoryErrors in Python, though I guess C++ is less often used from an
event loop).
The point is to provide the user with the stack trace of exactly where
you ran out of memory. Even if you can't follow it deep into C++-land,
the Python-side stack trace can be a worth a lot.
Note that I think (I just skimmed gcc documentation) the abort will
happen the moment you pop back to a C function (unless you compile the C
code with -fexceptions). I.e. once you're back in the CPython eval loop
you die, it doesn't propagate back to main(). That means you could crash
in a lot of callback cases too; your expectation of some kind of
propagation for callbacks is only true if there's C++ all the way
(according to my skimming, anyway).
Dag
There is a lot of C++ code out there using exceptions. I'd guess that
both
mathematical code and Google-written code is unlike most C++ code out
there
:-) Many C++ programmers go on and on about RAII and auto_ptrs and so
on,
and that doesn't have much point unless you throw an exception now
and then
(OK, there's the occasional return statement where it matters well).
True, I've seen a small subset of the C++ code
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
Den 03.07.2012 20:43, skrev Dag Sverre Seljebotn: Except for the fact that any code touching "new" could be raising exceptions? That propagates. There is a lot of C++ code out there using exceptions. I'd guess that both mathematical code and Google-written code is unlike most C++ code out there :-) Many C++ programmers go on and on about RAII and auto_ptrs and so on, and that doesn't have much point unless you throw an exception now and then (OK, there's the occasional return statement where it matters well). Usually there is just one C++ exception to care about: std::bad_alloc. It is important to know that it can be raised almost anywhere in C++ code that are using the STL. The majority of C++ programs never attempt to catch std::bad_alloc. Instead a program will set a global handler routine using std::set_new_handler. Usually a callback for memory failure will just display an error message and call std::exit with status std::EXIT_FAILURE. || When interfacing from Python I think a std::bad_alloc should be translated to a MemoryError exception if possible. Though others might argue that one should never try to recover from a memory failure. Arguably, recovering std::bad_alloc might not be possible if the heap is exhausted. Sturla ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
Sturla Molden, 04.07.2012 02:58: > Den 03.07.2012 20:43, skrev Dag Sverre Seljebotn: >> Except for the fact that any code touching "new" could be raising >> exceptions? That propagates. >> >> There is a lot of C++ code out there using exceptions. I'd guess that >> both mathematical code and Google-written code is unlike most C++ code >> out there :-) Many C++ programmers go on and on about RAII and auto_ptrs >> and so on, and that doesn't have much point unless you throw an exception >> now and then (OK, there's the occasional return statement where it >> matters well). > > Usually there is just one C++ exception to care about: std::bad_alloc. It > is important to know that it can be raised almost anywhere in C++ code that > are using the STL. > > The majority of C++ programs never attempt to catch std::bad_alloc. Instead > a program will set a global handler routine using std::set_new_handler. Interesting. Doesn't make much sense to do that in Cython, though (and users wouldn't like it if they intend to use it themselves). > Usually a callback for memory failure will just display an error message > and call std::exit with status std::EXIT_FAILURE. || > > When interfacing from Python I think a std::bad_alloc should be translated > to a MemoryError exception if possible. Though others might argue that one > should never try to recover from a memory failure. Arguably, recovering > std::bad_alloc might not be possible if the heap is exhausted. Usually, the intention is more to exit gracefully, i.e. to pass through all finally clauses, than to actually handle the error and continue. And I would expect that going back along that path would eventually free up some memory, so that this should actually work in most cases. Also, the allocation may have failed on a larger block of memory, which is then unused when the exception gets raised and can be used by cleanup code. I really don't think the world is all that dark here. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
