Re: [Python-Dev] working with Python threads from C extension module?
On Fri, 2007-09-07 at 16:20 -0700, Bill Janssen wrote:
> > #define SSL_ALLOW_THREADS {if (_ssl_locks != NULL) { Py_BEGIN_ALLOW_THREADS
> > }}
> > #define SSL_DISALLOW_THREADS {if (_ssl_locks != NULL) {
> > Py_BEGIN_ALLOW_THREADS }}
>
> I'd forgotten how convoluted Py_BEGIN_ALLOW_THREADS and
> Py_END_ALLOW_THREADS were. Anyone have any other suggestions about
> how to do this?
Be convoluted yourself and do this:
#define PySSL_BEGIN_ALLOW_THREADS { if (_ssl_locks) { Py_BEGIN_ALLOW_THREADS
#define PySSL_END_ALLOW_THREADS Py_END_ALLOW_THREADS } }
(Untested, but I think it should work.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Buildbot upgraded to 0.7.5
I just upgraded the buildbot master to 0.7.5. If you see any problems, please let me know. Neal: buildbot now supports reloading of configurations, without interrupting builds. Try "buildbot reconfig" when you make a change (certain changes would still require a restart). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] working with Python threads from C extension module?
> Be convoluted yourself and do this:
>
> #define PySSL_BEGIN_ALLOW_THREADS { if (_ssl_locks) { Py_BEGIN_ALLOW_THREADS
> #define PySSL_END_ALLOW_THREADS Py_END_ALLOW_THREADS } }
>
> (Untested, but I think it should work.)
Yes, that had occurred to me. We want the code inside the braces
still to run if the locks aren't held, so something more like
#define PySSL_BEGIN_ALLOW_THREADS { \
PyThreadState *_save; \
if (_ssl_locks_count>0) {_save = PyEval_SaveThread();}
#define PySSL_BLOCK_THREADS if
(_ssl_locks_count>0){PyEval_RestoreThread(_save)};
#define PySSL_UNBLOCK_THREADS if (_ssl_locks_count>0){_save =
PyEval_SaveThread()};
#define PySSL_END_ALLOW_THREADS if
(_ssl_locks_count>0){PyEval_RestoreThread(_save);} \
}
would do the trick. Unfortunately, this doesn't deal with the macro
behaviour. The user has "turned on" threading; they expect reads and
writes to yield the GIL so that other threads can make progress. But
the fact that threading has been "turned on" after the SSL module has
been initialized, means that threads don't work inside the SSL code.
So the user's understanding of the system will be broken.
No, I don't see any good way to fix this except to add a callback
chain inside PyThread_init_thread, which is run down when threads are
initialized. Any module which needs to set up threads registers itself
on that chain, and gets called as part of PyThread_init_thread. But
I'm far from the smartest person on this list :-), so perhaps someone
else will see a good solution.
This has got to be a problem with other extension modules linked to
libraries which have their own threading abstractions.
Bill
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] working with Python threads from C extension module?
On 08/09/2007, Bill Janssen <[EMAIL PROTECTED]> wrote:
>
> > Be convoluted yourself and do this:
> >
> > #define PySSL_BEGIN_ALLOW_THREADS { if (_ssl_locks) {
> Py_BEGIN_ALLOW_THREADS
> > #define PySSL_END_ALLOW_THREADS Py_END_ALLOW_THREADS } }
> >
> > (Untested, but I think it should work.)
>
> Yes, that had occurred to me. We want the code inside the braces
> still to run if the locks aren't held, so something more like
>
> #define PySSL_BEGIN_ALLOW_THREADS { \
> PyThreadState *_save; \
> if (_ssl_locks_count>0) {_save =
> PyEval_SaveThread();}
> #define PySSL_BLOCK_THREADS if
> (_ssl_locks_count>0){PyEval_RestoreThread(_save)};
> #define PySSL_UNBLOCK_THREADS if (_ssl_locks_count>0){_save =
> PyEval_SaveThread()};
> #define PySSL_END_ALLOW_THREADS if
> (_ssl_locks_count>0){PyEval_RestoreThread(_save);} \
> }
>
> would do the trick. Unfortunately, this doesn't deal with the macro
> behaviour. The user has "turned on" threading; they expect reads and
> writes to yield the GIL so that other threads can make progress. But
> the fact that threading has been "turned on" after the SSL module has
> been initialized, means that threads don't work inside the SSL code.
> So the user's understanding of the system will be broken.
>
> No, I don't see any good way to fix this except to add a callback
> chain inside PyThread_init_thread, which is run down when threads are
> initialized. Any module which needs to set up threads registers itself
> on that chain, and gets called as part of PyThread_init_thread. But
> I'm far from the smartest person on this list :-), so perhaps someone
> else will see a good solution.
I think this is a helpful additional tool to solve threading problems.
Doesn't solve everything, but it certainly helps :-)
For instance, one thing it doesn't solve is when a library being wrapped can
be initialized with multithreading support, but only allows such
initialization as a very first API call; you can't initialize threading at
any arbitrary time during application runtime. Unfortunately I don't think
there is any sane way to fix this problem :-(
This has got to be a problem with other extension modules linked to
> libraries which have their own threading abstractions.
Yes.
Another problem is that python extensions may not wish to incur performance
penalty of python threading calls. For instance, pyorbit has these macros:
#define pyorbit_gil_state_ensure() (PyEval_ThreadsInitialized()?
(PyGILState_Ensure()) : 0)
#define pyorbit_gil_state_release(state) G_STMT_START { \
if (PyEval_ThreadsInitialized())\
PyGILState_Release(state); \
} G_STMT_END
#define pyorbit_begin_allow_threads \
G_STMT_START { \
PyThreadState *_save = NULL;\
if (PyEval_ThreadsInitialized())\
_save = PyEval_SaveThread();
#define pyorbit_end_allow_threads \
if (PyEval_ThreadsInitialized())\
PyEval_RestoreThread(_save);\
} G_STMT_END
They all call PyEval_ThreadsInitialized() before doing anything thread
related to save some performance. The other reason to do it this way is
that the Python API calls themselves abort if they are called with threading
not initialized. It would be nice the upstream python GIL macros were more
like pyorbit and became no-ops when threading is not enabled.
--
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] which SSL client protocols work with which server protocols?
I've now built a framework in test_ssl to test all client protocols (SSL2, SSL3, SSL23, TLS1) against all server protocols, and here's what I've come up with. Servers are along the X axis, and clients are on the Y axis. "Yes" means that that client protocol can talk to that server protocol. SSL2SSL3SS23TLS1 SSL2yes no no no SSL3yes yes yes no SSL23 no no yes no TLS1no no yes yes I'm a bit surprised by the facts that (1) an SSL2 client can't connect to an SSL23 server, and (2) an SSL23 client can *only* connect to an SSL23 server. Can anyone verify that these combos (the results of testing with the Python framework) are indeed to be expected? Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] working with Python threads from C extension module?
> This has got to be a problem with other extension modules linked to > libraries which have their own threading abstractions. Sure enough, sqlite3 simply assumes threads (won't build without them), and turns them on if it's used (by calling PyThread_get_thread_ident(), which in turn calls PyThread_init_thread()). Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] testing in a Python --without-threads build
I can't seem to run the regression tests in a --without-threads build. Might be interesting to configure a buildbot this way to keep ourselves honest. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] testing in a Python --without-threads build
> I can't seem to run the regression tests in a --without-threads build. > Might be interesting to configure a buildbot this way to keep > ourselves honest. Because regrtest.py was importing test_socket_ssl without catching the ImportError exception: % ./python.exe ./Lib/test/regrtest.py test_socket_ssl test_socket_ssl test_socket_ssl skipped -- No module named thread 1 test skipped: test_socket_ssl Traceback (most recent call last): File "./Lib/test/regrtest.py", line 1190, in main() File "./Lib/test/regrtest.py", line 416, in main e = _ExpectedSkips() File "./Lib/test/regrtest.py", line , in __init__ from test import test_socket_ssl File "/local/python/trunk/src/Lib/test/test_socket_ssl.py", line 8, in import threading File "/local/python/trunk/src/Lib/threading.py", line 6, in import thread ImportError: No module named thread % So, is this an "expected skip" or not? Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] testing in a Python --without-threads build
>> I can't seem to run the regression tests in a --without-threads build. >> Might be interesting to configure a buildbot this way to keep >> ourselves honest. > > Because regrtest.py was importing test_socket_ssl without catching the > ImportError exception: If that is the reason you cannot run it, then it seems it works just fine. There is nothing wrong with tests getting skipped. > So, is this an "expected skip" or not? No. IIUC, "expected skips" are a platform property. For your platform, support for threads is expected (whatever your platform is as log as it was built in this millenium). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] testing in a Python --without-threads build
> > Because regrtest.py was importing test_socket_ssl without catching the
> > ImportError exception:
>
> If that is the reason you cannot run it, then it seems it works just
> fine. There is nothing wrong with tests getting skipped.
It wasn't getting skipped, it was crashing the regression testing harness.
test_unittest catches the ImportError, but this was imported directly
from regrtest.py.
> > So, is this an "expected skip" or not?
>
> No. IIUC, "expected skips" are a platform property. For your platform,
> support for threads is expected (whatever your platform is as log as
> it was built in this millenium).
OK. I'll put in a check for this. In fact, here's a patch:
Index: Lib/test/regrtest.py
===
--- Lib/test/regrtest.py(revision 58052)
+++ Lib/test/regrtest.py(working copy)
@@ -1108,7 +1108,6 @@
class _ExpectedSkips:
def __init__(self):
import os.path
-from test import test_socket_ssl
from test import test_timeout
self.valid = False
@@ -1122,8 +1121,13 @@
if not os.path.supports_unicode_filenames:
self.expected.add('test_pep277')
-if test_socket_ssl.skip_expected:
-self.expected.add('test_socket_ssl')
+try:
+from test import test_socket_ssl
+except ImportError:
+pass
+else:
+if test_socket_ssl.skip_expected:
+self.expected.add('test_socket_ssl')
if test_timeout.skip_expected:
self.expected.add('test_timeout')
Bill
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] what platforms require RAND_add() before using SSL?
There are some functions in _ssl.c for gathering randomness from a daemon, and adding that randomness to the pseudo-random number generator in SSL, before using SSL. There's a note there saying that "on some platform" this is necessary. Anyone know which platforms? Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] 3.0 crypto
> We're already linking against the OpenSSL EVP libraries for hashlib
> (and against the OpenSSL SSL libraries for the SSL support). It
> wouldn't be hard to expose the EVP functions a bit more, essentially
> as hash functions that return long (and reversible) hashes:
>
>encryptor = opensslevp.encryptor("AES-256-CBC", ...maybe some options...)
>encryptor.update(...some plaintext...)
Almost certainly this signature should be
encryptor = opensslevp.encryptor("AES-256-CBC", KEY, ...options...)
and correspondingly
decryptor = opensslevp.decryptor("AES-256-CBC", KEY, ...options...)
Bill
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] can't run test_tcl remotely logged in on an OS X machine
"test_tcl" fails on me (OS X 10.4.10 on an Intel Mac, remotely logged in via SSH and X Windows): % test_tcl 2007-09-08 17:00:22.629 python.exe[4163] CFLog (0): CFMessagePort: bootstrap_register(): failed 1100 (0x44c), port = 0x3a03, name = 'Processes-0.58327041' See /usr/include/servers/bootstrap_defs.h for the error codes. 2007-09-08 17:00:22.630 python.exe[4163] CFLog (99): CFMessagePortCreateLocal(): failed to name Mach port (Processes-0.58327041) CFMessagePortCreateLocal failed (name = Processes-0.58327041 error = 0) Abort % This is on the trunk. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] testing in a Python --without-threads build
Might expected skips instead be based on your current configuration instead of what someone statically decided what would be appropriate for your platform? Every new release I have to go through the 'unexpected skips' to determine that they're perfectly fine for how I configured python. It seems that we ought to provide a mechanism for querying python for how the build was configured (although for non-unittest cases, failing to import some modules is usually sufficient information - knowing why they fail probably doesn't matter) On 9/8/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > >> I can't seem to run the regression tests in a --without-threads build. > >> Might be interesting to configure a buildbot this way to keep > >> ourselves honest. > > > > Because regrtest.py was importing test_socket_ssl without catching the > > ImportError exception: > > If that is the reason you cannot run it, then it seems it works just > fine. There is nothing wrong with tests getting skipped. > > > So, is this an "expected skip" or not? > > No. IIUC, "expected skips" are a platform property. For your platform, > support for threads is expected (whatever your platform is as log as > it was built in this millenium). > > Regards, > Martin > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/nick.bastin%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
