Hi Python experts,
I need to extend Python with C/C++ to interact with an in-house legacy API. I'm
implementing a data subscriber with Python, which subscribes to a data
publisher (actually a ZeroMQ publisher socket) and will get notified once any
new messages are fed. In my subscriber, the messages are dumped to a data
processor after received. The subscriber will also get notified by the
processor when done. Since the data processor is written in C++, I have to
extend the Python code with a simple C++ module.
Below is a simplified runnable code sample of my data subscriber. Code main.py
, in which the module proc represents the processor, subscribe to a ZeroMQ
socket on localhost:10000 , setup the callback, and send the received message
to the processor by calling proc.onMsg .
#!/bin/python
# main.py
import gevent
import logging
import zmq.green as zmq
import pub
import proc
logging.basicConfig( format='[%(levelname)s] %(message)s', level=logging.DEBUG )
SUB_ADDR = 'tcp://localhost:10000'
def setupMqAndReceive():
'''Setup the message queue and receive messages.
'''
ctx = zmq.Context()
sock = ctx.socket( zmq.SUB )
# add topics
sock.setsockopt_string( zmq.SUBSCRIBE, 'Hello' )
sock.connect( SUB_ADDR )
while True:
msg = sock.recv().decode( 'utf-8' )
proc.onMsg( msg )
def callback( a, b ):
print( '[callback]', a, b )
def main():
'''Entrance of the module.
'''
pub.start()
proc.setCallback( callback )
'''A simple on-liner
gevent.spawn( setupMqAndReceive ).join()
works. However, the received messages will not be
processed by the processor.
'''
gevent.spawn( setupMqAndReceive )
proc.start()
Module proc is simplified with three functions exported:
* setCallback setup the callback function, so that when the message
processed, my subscriber could be notified;
* onMsg is invoked by the subscriber;
* start setups a new worker thread to handle the messages from subscriber
and make the main thread join to wait for the worker thread to exit.
The full version of source code could be found on github at
https://github.com/more-more-tea/python_gil . Nevertheless, it does not run as
my expectation. Once the processor thread is added, the subscriber cannot
receive data from the publisher in the gevent loop. If I simply drop the data
processor module, the subscriber gevent loop could receive the messages from
the publisher. The same behavior could be observed with threading.Thread
instead of gevent.
I posted the same question on StackOverflow
http://stackoverflow.com/questions/37513957/does-python-gil-need-to-be-taken-care-when-work-with-multi-thread-c-extension
with one answer direct me to the article
https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock
I read the document but still not sure where to add the
PyGILState_Ensure/PyGILState_Release. Without a running sample, it is a little
bit hard to understand how Python thread and the native pthread interact.
Would you mind review the source code and direct the right way to make it
running?
Thank you very much for your time and best regards!
Shuang
--
https://mail.python.org/mailman/listinfo/python-list
