Re: [Python-Dev] cffi in stdlib

2013-12-17 Thread Stefan Krah
Maciej Fijalkowski  wrote:
> I would like to discuss on the language summit a potential inclusion
> of cffi[1] into stdlib. This is a project Armin Rigo has been working
> for a while, with some input from other developers.

I've tried cffi (admittedly only in a toy script) and find it very nice
to use.

Here's a comparison (pi benchmark) between wrapping libmpdec using a
C-extension (_decimal), cffi and ctypes:


+---+--+--+-+
|   | _decimal |  ctypes  |   cffi  |
+===+==+==+=+
| cpython-tip (with-system-ffi) |   0.19s  |   5.40s  |  5.14s  |
+---+--+--+-+
| cpython-2.7 (with-system-ffi) |n/a   |   4.46s  |  5.18s  |
+---+--+--+-+
|  Ubuntu-cpython-2.7   |n/a   |   3.63s  |-|
+---+--+--+-+
|  pypy-2.2.1-linux64   |n/a   |  125.9s  |  0.94s  |
+---+--+--+-+
| pypy3-2.1-beta1-linux64   |n/a   |  264.9s  |  2.93s  |
+---+--+--+-+


I guess the key points are that C-extensions are hard to beat and that
cffi performance on pypy-2 is outstanding. Additionally it's worth noting
that Ubuntu does something in their Python build that we should do, too.


+1 for cffi in the stdlib.



Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] cffi in stdlib

2013-12-17 Thread Brett Cannon
Maybe someone from PyPy should bring this up as an official topic at the
language summit to figure out the blockers (again). Or it can join regex on
the list of "module discussed for addition at the language summit but never
quite pushed to commitment". =)


On Tue, Dec 17, 2013 at 11:43 AM, Stefan Krah  wrote:

> Maciej Fijalkowski  wrote:
> > I would like to discuss on the language summit a potential inclusion
> > of cffi[1] into stdlib. This is a project Armin Rigo has been working
> > for a while, with some input from other developers.
>
> I've tried cffi (admittedly only in a toy script) and find it very nice
> to use.
>
> Here's a comparison (pi benchmark) between wrapping libmpdec using a
> C-extension (_decimal), cffi and ctypes:
>
>
> +---+--+--+-+
> |   | _decimal |  ctypes  |   cffi  |
> +===+==+==+=+
> | cpython-tip (with-system-ffi) |   0.19s  |   5.40s  |  5.14s  |
> +---+--+--+-+
> | cpython-2.7 (with-system-ffi) |n/a   |   4.46s  |  5.18s  |
> +---+--+--+-+
> |  Ubuntu-cpython-2.7   |n/a   |   3.63s  |-|
> +---+--+--+-+
> |  pypy-2.2.1-linux64   |n/a   |  125.9s  |  0.94s  |
> +---+--+--+-+
> | pypy3-2.1-beta1-linux64   |n/a   |  264.9s  |  2.93s  |
> +---+--+--+-+
>
>
> I guess the key points are that C-extensions are hard to beat and that
> cffi performance on pypy-2 is outstanding. Additionally it's worth noting
> that Ubuntu does something in their Python build that we should do, too.
>
>
> +1 for cffi in the stdlib.
>
>
>
> Stefan Krah
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] Backward-incompatible change to random.randrange in 2.7.6

2013-12-17 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

http://hg.python.org/cpython/rev/b1e94e332ec8


Do we really want to change an undocumented-but-effectively-public API in
a late-in-the-release-cycle third dot release?  It caused, ZODB's tests
to fail, for instance.

While the docstring said, "Don't use the 'int', 'default', and 'maxwidth'
arguments", their names were not intrinsically private.  In particular,
passing in the 'int' argument was a strategy for generating compatible
long values when straddling Python 2.x / Python 3.x.




Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKwlXAACgkQ+gerLs4ltQ60DQCgzlO8mHMXQ0vsHNpS9GKwjpmD
G6oAoMIjtrKkGTlFj0b9Tfdj5BCu1rYS
=GxuS
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] cffi in stdlib

2013-12-17 Thread Maciej Fijalkowski
On Tue, Dec 17, 2013 at 7:21 PM, Brett Cannon  wrote:
> Maybe someone from PyPy should bring this up as an official topic at the
> language summit to figure out the blockers (again). Or it can join regex on
> the list of "module discussed for addition at the language summit but never
> quite pushed to commitment". =)

we're still working on resolving discussed issues before officially
proposing it for inclusion.

>
>
> On Tue, Dec 17, 2013 at 11:43 AM, Stefan Krah  wrote:
>>
>> Maciej Fijalkowski  wrote:
>> > I would like to discuss on the language summit a potential inclusion
>> > of cffi[1] into stdlib. This is a project Armin Rigo has been working
>> > for a while, with some input from other developers.
>>
>> I've tried cffi (admittedly only in a toy script) and find it very nice
>> to use.
>>
>> Here's a comparison (pi benchmark) between wrapping libmpdec using a
>> C-extension (_decimal), cffi and ctypes:
>>
>>
>> +---+--+--+-+
>> |   | _decimal |  ctypes  |   cffi  |
>> +===+==+==+=+
>> | cpython-tip (with-system-ffi) |   0.19s  |   5.40s  |  5.14s  |
>> +---+--+--+-+
>> | cpython-2.7 (with-system-ffi) |n/a   |   4.46s  |  5.18s  |
>> +---+--+--+-+
>> |  Ubuntu-cpython-2.7   |n/a   |   3.63s  |-|
>> +---+--+--+-+
>> |  pypy-2.2.1-linux64   |n/a   |  125.9s  |  0.94s  |
>> +---+--+--+-+
>> | pypy3-2.1-beta1-linux64   |n/a   |  264.9s  |  2.93s  |
>> +---+--+--+-+
>>
>>
>> I guess the key points are that C-extensions are hard to beat and that
>> cffi performance on pypy-2 is outstanding. Additionally it's worth noting
>> that Ubuntu does something in their Python build that we should do, too.
>>
>>
>> +1 for cffi in the stdlib.
>>
>>
>>
>> Stefan Krah
>>
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6

2013-12-17 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 12/17/2013 01:40 PM, Guido van Rossum wrote:
> This really seems a case of ZODB depending on internals where it 
> really, really should have known better. Calling this "a de-facto 
> public interface" seems way too far a stretch of the intention. And 
> please don't fix it by version-testing and using a different argument 
> name...

The usage is *ancient*:  Jeremy checked it in 2001-10-05:


https://github.com/zopefoundation/ZODB/commit/fd1653580ca67bdc281fb8c54662c97dd3cf1aaa

The comment about "do not pass the 'int' or 'default' arguments" goes
back to at least the 'whrandom.py' module in 1.5.2. ;)



Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlKwn7QACgkQ+gerLs4ltQ4WzACeOMXqg5Jg8ck3vK3cCDuTgKSS
8GwAn0yR8ukdQTh5Wo0jGDHq/AIgu+Yg
=fTUf
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6

2013-12-17 Thread Barry Warsaw
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On Dec 17, 2013, at 01:18 PM, Tres Seaver wrote:

>http://hg.python.org/cpython/rev/b1e94e332ec8
>
>Do we really want to change an undocumented-but-effectively-public API in
>a late-in-the-release-cycle third dot release?  It caused, ZODB's tests
>to fail, for instance.
>
>While the docstring said, "Don't use the 'int', 'default', and 'maxwidth'
>arguments", their names were not intrinsically private.  In particular,
>passing in the 'int' argument was a strategy for generating compatible
>long values when straddling Python 2.x / Python 3.x.

Being quite sensitive to complaints about API breaks in patch releases, it
does seem like a bad idea to change these in a backward incompatible way.

It's true the docstring warns against it, and that the module does not
document those arguments, but they're still there and changing them breaks
existing code.  That should be enough to revert and rewrite the change.

I don't think the API *has* to change in a backward incompatible way either.
The methods could be given **kws with a bit of hackery to figure out whether
the old API was being used (keys: int, default, maxwidth) or the new API was
being used (keys: _int and _maxwidth).  Yeah it's ugly, but we serve our users
better by doing it that way.

Cheers,
- -Barry
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.15 (GNU/Linux)

iQIcBAEBCAAGBQJSsJ4yAAoJEBJutWOnSwa/7EsP/1P3VaaeWdLnWkQKa06e4TpY
oXHG3YUMAQ4AjrBFTc5KL1GQtSkg1m6pQKZf937XYhi9d4xfwbaSBqCBWYPkNdrx
DR2Ix0Y/HwVPQvlqgfLWPS6TaZIrA2ssCVoOgWL2kmX+5KugIUy3O521OOre14jo
jCQZTOL2sloT+/TlX1PSImYRrJnK9yklW0CW13OVnj5BsvjzQPtsL0z1ueSrebqu
awTRCI7O8gpBcw7vcXnB6ZAfuA4urLQ+AnBXF1p5Rsxs3tGW9HS7L+QYWBM7sktV
Hmb5ZIdH+4+gkCws2LSowt+06M3WEyLkGhwOH1gNU3WTgywcJ4L9sh7WwLZN5eCe
Wz37LQ1fVFjpJhX2zOtI6djqL5r+6xpBxtuig0ezNJbAc1mhhvkPkTjMyvXqQzwX
HJQ7OMthiHqVnNJRZQZs4tSeCPgRDkDSNe/RWSyh6gYr6Gn6wwDcbW20RrbbbU4y
eKyY2VHQ0MKD283HVb3nfgelN96OgqWbpG/uk6mRPwPU1oIUyDDuhyyzNabggTSV
n97lPsHvOJyPehKdu+QKxpFULlX6KEmTCLJsUIWsjMCVIUnHjobkpOqW20KveLr+
cPcSHDGIPKv4qnFJuihYWBz9NPbUT2xebaP02bL7Wu5UVDFxQ3t0P/wuhLoNjRzU
vkUJ8m7CnRDFGuGPy6NQ
=lw3q
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6

2013-12-17 Thread Guido van Rossum
This really seems a case of ZODB depending on internals where it
really, really should have known better. Calling this "a de-facto
public interface" seems way too far a stretch of the intention. And
please don't fix it by version-testing and using a different argument
name...

On Tue, Dec 17, 2013 at 10:18 AM, Tres Seaver  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> http://hg.python.org/cpython/rev/b1e94e332ec8
>
>
> Do we really want to change an undocumented-but-effectively-public API in
> a late-in-the-release-cycle third dot release?  It caused, ZODB's tests
> to fail, for instance.
>
> While the docstring said, "Don't use the 'int', 'default', and 'maxwidth'
> arguments", their names were not intrinsically private.  In particular,
> passing in the 'int' argument was a strategy for generating compatible
> long values when straddling Python 2.x / Python 3.x.
>
>
>
>
> Tres.
> - --
> ===
> Tres Seaver  +1 540-429-0999  tsea...@palladion.com
> Palladion Software   "Excellence by Design"http://palladion.com
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlKwlXAACgkQ+gerLs4ltQ60DQCgzlO8mHMXQ0vsHNpS9GKwjpmD
> G6oAoMIjtrKkGTlFj0b9Tfdj5BCu1rYS
> =GxuS
> -END PGP SIGNATURE-
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6

2013-12-17 Thread Tim Peters
[Barry]
> ...
> I don't think the API *has* to change in a backward incompatible way either.
> The methods could be given **kws with a bit of hackery to figure out whether
> the old API was being used (keys: int, default, maxwidth) or the new API was
> being used (keys: _int and _maxwidth).  Yeah it's ugly, but we serve our users
> better by doing it that way.

-1.  The speed of randrange() is crucial for many applications;
indeed, that's the only reason it abused default arguments to begin
with (i.e., to make `int` et alia fast local lookups).  Digging
through a dict to guess which API was invoked would ruin that.

And the ZODB tests in question deserve whatever they get ;-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] cffi in stdlib

2013-12-17 Thread Nick Coghlan
On 18 Dec 2013 06:21, "Maciej Fijalkowski"  wrote:
>
> On Tue, Dec 17, 2013 at 7:21 PM, Brett Cannon  wrote:
> > Maybe someone from PyPy should bring this up as an official topic at the
> > language summit to figure out the blockers (again). Or it can join
regex on
> > the list of "module discussed for addition at the language summit but
never
> > quite pushed to commitment". =)
>
> we're still working on resolving discussed issues before officially
> proposing it for inclusion.

Note that there's also now a link chain from the CPython extension creation
docs to cffi (and Cython) - the cross version Python Packaging User Guide
now has a section on binary extensions that covers several of the
alternatives to writing them by hand, while the stdlib extension writing
guide has a note at the beginning pointing to that resource.

Cheers,
Nick.

>
> >
> >
> > On Tue, Dec 17, 2013 at 11:43 AM, Stefan Krah 
wrote:
> >>
> >> Maciej Fijalkowski  wrote:
> >> > I would like to discuss on the language summit a potential inclusion
> >> > of cffi[1] into stdlib. This is a project Armin Rigo has been working
> >> > for a while, with some input from other developers.
> >>
> >> I've tried cffi (admittedly only in a toy script) and find it very nice
> >> to use.
> >>
> >> Here's a comparison (pi benchmark) between wrapping libmpdec using a
> >> C-extension (_decimal), cffi and ctypes:
> >>
> >>
> >> +---+--+--+-+
> >> |   | _decimal |  ctypes  |   cffi  |
> >> +===+==+==+=+
> >> | cpython-tip (with-system-ffi) |   0.19s  |   5.40s  |  5.14s  |
> >> +---+--+--+-+
> >> | cpython-2.7 (with-system-ffi) |n/a   |   4.46s  |  5.18s  |
> >> +---+--+--+-+
> >> |  Ubuntu-cpython-2.7   |n/a   |   3.63s  |-|
> >> +---+--+--+-+
> >> |  pypy-2.2.1-linux64   |n/a   |  125.9s  |  0.94s  |
> >> +---+--+--+-+
> >> | pypy3-2.1-beta1-linux64   |n/a   |  264.9s  |  2.93s  |
> >> +---+--+--+-+
> >>
> >>
> >> I guess the key points are that C-extensions are hard to beat and that
> >> cffi performance on pypy-2 is outstanding. Additionally it's worth
noting
> >> that Ubuntu does something in their Python build that we should do,
too.
> >>
> >>
> >> +1 for cffi in the stdlib.
> >>
> >>
> >>
> >> Stefan Krah
> >>
> >>
> >>
> >> ___
> >> Python-Dev mailing list
> >> Python-Dev@python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/brett%40python.org
> >
> >
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
> >
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6

2013-12-17 Thread Antoine Pitrou
On Tue, 17 Dec 2013 13:18:25 -0500
Tres Seaver  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> http://hg.python.org/cpython/rev/b1e94e332ec8
> 
> 
> Do we really want to change an undocumented-but-effectively-public API in
> a late-in-the-release-cycle third dot release?  It caused, ZODB's tests
> to fail, for instance.

Given the change doesn't seem to bring any visible change for users
(either performance or robustness), I think it would be safe to revert
it *in 2.7*.

Of course Zope's practice is still rather bad here... 

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6

2013-12-17 Thread Benjamin Peterson
2013/12/17 Antoine Pitrou :
> On Tue, 17 Dec 2013 13:18:25 -0500
> Tres Seaver  wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> http://hg.python.org/cpython/rev/b1e94e332ec8
>>
>>
>> Do we really want to change an undocumented-but-effectively-public API in
>> a late-in-the-release-cycle third dot release?  It caused, ZODB's tests
>> to fail, for instance.
>
> Given the change doesn't seem to bring any visible change for users
> (either performance or robustness), I think it would be safe to revert
> it *in 2.7*.

I agree with Antoine. It's better not to break even morally-broken
programs like the zope tests in 2.7.x if it doesn't win anything.


-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] thread issues when embedding Python

2013-12-17 Thread Daniel Pocock


I've successfully embedded Python for a single thread

I tried to extend the implementation for multiple threads (a worker
thread scenario) and I'm encountering either deadlocks or seg faults
depending upon how I got about it.

There seems to be some inconsistency between what is covered in the docs
here:

http://docs.python.org/2/c-api/init.html#non-python-created-threads

and the experiences of other users trying the same thing, e.g.

http://bugs.python.org/issue19576
http://wiki.blender.org/index.php/Dev:2.4/Source/Python/API/Threads

Can anybody comment on the situation, in particular,

Is the non-python-created-threads documentation accurate for v2.7?

If a main thread does things like importing a module and obtaining a
reference to a Python method, can those things be used by other C++
threads or do they have to repeat those lookups?

Is there any logic that needs to be executed once only as each thread is
started? (the doc suggests just PyGILState_Ensure/PyGILState_Release
each time a thread accesses Python methods - is there anything else?)

Given the bug 19576, what is the most portable way to code this to work
reliably on unfixed Python versions?  (e.g. should users always
explicitly call PyEval_InitThreads() in their main thread or worker
threads or both?)




Here is my actual source code:

https://svn.resiprocate.org/viewsvn/resiprocate/main/repro/plugins/pyroute/

  (see example.py for a trivial example of what it does)

The problem that I encounter:

- the init stuff runs fine in PyRoutePlugin.cxx,
it calls Py_Initialize, PyEval_InitThreads, PyImport_ImportModule
  and looks up the "provide_route" method in the module
it creates a PyRouteWorker object,
  giving it a reference to "provide_route"
it creates a thread pool to run the worker

- the PyRouteWorker::process() method is invoked in one of those threads

- it crashes when trying to call the "provide_route" method
PyRouteWorker.cxx:
  routes = mAction.apply(args);


Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x730b8700 (LWP 23965)]
0x73d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
(gdb) bt
#0  0x73d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
#1  0x73d6b647 in PyEval_CallObjectWithKeywords ()
   from /usr/lib/libpython2.7.so.1.0
#2  0x7414885a in apply (args=..., this=)
at /usr/include/python2.7/CXX/Python2/Objects.hxx:3215
#3  repro::PyRouteWorker::process (this=0x6f00a0, msg=)
at PyRouteWorker.cxx:98
#4  0x77b879e1 in repro::WorkerThread::thread (this=0x68e110)
at WorkerThread.cxx:36
#5  0x770b7a2f in threadIfThreadWrapper (threadParm=)
at ThreadIf.cxx:51
#6  0x765ffb50 in start_thread (arg=)
at pthread_create.c:304
#7  0x75999a7d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8  0x in ?? ()
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] thread issues when embedding Python

2013-12-17 Thread Daniel Pocock


I've successfully embedded Python for a single thread

I tried to extend the implementation for multiple threads (a worker
thread scenario) and I'm encountering either deadlocks or seg faults
depending upon how I got about it.

There seems to be some inconsistency between what is covered in the docs
here:

http://docs.python.org/2/c-api/init.html#non-python-created-threads

and the experiences of other users trying the same thing, e.g.

http://bugs.python.org/issue19576
http://wiki.blender.org/index.php/Dev:2.4/Source/Python/API/Threads

Can anybody comment on the situation, in particular,

Is the non-python-created-threads documentation accurate for v2.7?

If a main thread does things like importing a module and obtaining a
reference to a Python method, can those things be used by other C++
threads or do they have to repeat those lookups?

Is there any logic that needs to be executed once only as each thread is
started? (the doc suggests just PyGILState_Ensure/PyGILState_Release
each time a thread accesses Python methods - is there anything else?)

Given the bug 19576, what is the most portable way to code this to work
reliably on unfixed Python versions?  (e.g. should users always
explicitly call PyEval_InitThreads() in their main thread or worker
threads or both?)




Here is my actual source code:

https://svn.resiprocate.org/viewsvn/resiprocate/main/repro/plugins/pyroute/

  (see example.py for a trivial example of what it does)

The problem that I encounter:

- the init stuff runs fine in PyRoutePlugin.cxx,
it calls Py_Initialize, PyEval_InitThreads, PyImport_ImportModule
  and looks up the "provide_route" method in the module
it creates a PyRouteWorker object,
  giving it a reference to "provide_route"
it creates a thread pool to run the worker

- the PyRouteWorker::process() method is invoked in one of those threads

- it crashes when trying to call the "provide_route" method
PyRouteWorker.cxx:
  routes = mAction.apply(args);


Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x730b8700 (LWP 23965)]
0x73d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
(gdb) bt
#0  0x73d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
#1  0x73d6b647 in PyEval_CallObjectWithKeywords ()
   from /usr/lib/libpython2.7.so.1.0
#2  0x7414885a in apply (args=..., this=)
at /usr/include/python2.7/CXX/Python2/Objects.hxx:3215
#3  repro::PyRouteWorker::process (this=0x6f00a0, msg=)
at PyRouteWorker.cxx:98
#4  0x77b879e1 in repro::WorkerThread::thread (this=0x68e110)
at WorkerThread.cxx:36
#5  0x770b7a2f in threadIfThreadWrapper (threadParm=)
at ThreadIf.cxx:51
#6  0x765ffb50 in start_thread (arg=)
at pthread_create.c:304
#7  0x75999a7d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8  0x in ?? ()
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6

2013-12-17 Thread Donald Stufft
Isn't changing it in 2.7.6 which is already released and then reverting in 
2.7.7 worse? Either way 2.7.6 will have this change and be in the wild and 
broken for people who depend on it

> On Dec 17, 2013, at 5:54 PM, Benjamin Peterson  wrote:
> 
> 2013/12/17 Antoine Pitrou :
>> On Tue, 17 Dec 2013 13:18:25 -0500
>> Tres Seaver  wrote:
>>> -BEGIN PGP SIGNED MESSAGE-
>>> Hash: SHA1
>>> 
>>> http://hg.python.org/cpython/rev/b1e94e332ec8
>>> 
>>> 
>>> Do we really want to change an undocumented-but-effectively-public API in
>>> a late-in-the-release-cycle third dot release?  It caused, ZODB's tests
>>> to fail, for instance.
>> 
>> Given the change doesn't seem to bring any visible change for users
>> (either performance or robustness), I think it would be safe to revert
>> it *in 2.7*.
> 
> I agree with Antoine. It's better not to break even morally-broken
> programs like the zope tests in 2.7.x if it doesn't win anything.
> 
> 
> -- 
> Regards,
> Benjamin
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6

2013-12-17 Thread Benjamin Peterson
Mostly yes, but at least you could tell people to upgrade straight to
2.7.7 and skip 2.7.6.

2013/12/17 Donald Stufft :
> Isn't changing it in 2.7.6 which is already released and then reverting in 
> 2.7.7 worse? Either way 2.7.6 will have this change and be in the wild and 
> broken for people who depend on it
>
>> On Dec 17, 2013, at 5:54 PM, Benjamin Peterson  wrote:
>>
>> 2013/12/17 Antoine Pitrou :
>>> On Tue, 17 Dec 2013 13:18:25 -0500
>>> Tres Seaver  wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 http://hg.python.org/cpython/rev/b1e94e332ec8


 Do we really want to change an undocumented-but-effectively-public API in
 a late-in-the-release-cycle third dot release?  It caused, ZODB's tests
 to fail, for instance.
>>>
>>> Given the change doesn't seem to bring any visible change for users
>>> (either performance or robustness), I think it would be safe to revert
>>> it *in 2.7*.
>>
>> I agree with Antoine. It's better not to break even morally-broken
>> programs like the zope tests in 2.7.x if it doesn't win anything.
>>
>>
>> --
>> Regards,
>> Benjamin
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com