Hi,
I have implemented my thread as shown in uwsgi docs, somaro.py (
http://uwsgi-docs.readthedocs.org/en/latest/Mules.html )
The problem is mule workers constantly die in a few seconds. It seems like;
despite all try/catch, mule worker quits all of a sudden and quietly. And my
application tries to recreate listeners instead of the dead ones (which I setup
that way).
And it’s like they are fighting. Because the listener threads seem to die in a
few seconds.
Why do you think uwsgi kills the threads or threads die anyhow? I just couldn’t
find the reason at all.
I need 3 threads of background job must wait for a command. But they die.
uwsgi config myapp.ini:
-----------------------------------------------------------
[uwsgi]
gid = www-data
uid = www-data
socket=/var/run/myapplication.socket
chdir=/var/www-files/mypythonfiles/
master=true
module=index
processes=20
limit-as=256
callable=app
plugins=/usr/lib/uwsgi/plugins/python27_plugin.so,/usr/lib/uwsgi/plugins/spooler_plugin.so
logto2=loglog.log
vacuum=true
pythonpath=/var/www-files/mypythonfiles/
no-orphans=true
vhost=true
enable-threads=true
import=index
stats=127.0.0.1:1717
spooler-python-import=process
spooler=/var/www-files/mypythonfiles/spooldir/
spooler-chdir=/var/www-files/mypythonfiles/
spooler-processes=20
spooler-max-tasks=40
spooler-freq=20
spooler-harakiri=21600
# 6 hours harakiri
mule=listener.py
mule=anotherJobsListener2.py
mule=anotherJobsListener3.py
listener.py
-----------------------------------------------------------
def runNewListener():
""" New listener fork """
global THREAD_OBJS
t = DoJobThread()
t.daemon = True
t.start()
THREAD_OBJS.append(t)
def doAnnounceAlive():
""" Main loop. Will announce listeners into Redis. once a minute """
global THREAD_OBJS, MAX_LISTENER_COUNT, CHECK_LISTENER_FREQ
while True:
# make sure there are 3 listeners working at any time
try:
if len([t for t in THREAD_OBJS if not t.isAlive()]) >
0: # maybe better for memory rather than calling directly:
THREAD_OBJS = filter(lambda t: t != None and
t.isAlive() , THREAD_OBJS)
for i in range(0, 3-len(THREAD_OBJS)): runNewListener()
# fill empty listener quota
for t in THREAD_OBJS:
db.rds().setex('THREAD_BGJOB_Alive.%s' %
t.name, 120, datetime.today().strftime('%Y.%m.%d %H:%M:%S'))
except:
log.error('Failed to traverse listener thread
objects.', exc_info=True)
time.sleep(10)
if __name__ == '__main__':
doAnnounceAlive()
log.info(‘Background Job Mule on.')
job execution:
-----------------------------------------------------------
class DoJobThread(Thread):
def __init__(self):
log.info(‘Created new job thread’)
def run(self):
while True:
try:
do bla bla
catch:
log.critical(‘Listener died’, exc_info=True)
Logs are:
-----------------------------------------------------------
Background Job Mule on.
Created new job thread
Created new job thread
Created new job thread
Background Job Mule on.
Created new job thread
Created new job thread
Created new job thread
Created new job thread
Created new job thread
Created new job thread
Created new job thread
Created new job thread
Created new job thread
Created new job thread
Background Job Mule on.
...
On 10 Nov 2014 at 19:35:10, Dincer Kavraal ([email protected]) wrote:
Thank you for your intent to help. I will check.
Is there a way to check if mule is running? Or how many of them doing what?
--
Dincer Kavraal
Sent with Airmail
On 10 Nov 2014 at 20:53:41, Roberto De Ioris ([email protected]) wrote:
> I have a message processing job which listens to the message queue. When
> my
> flask application is initiated, I try to create 3 workers (or whatever
> called in that sense) of them working through @spoolraw.
>
> What is the best way to ensure a background task is running and ensure it
> is not killed by uwsgi? Is the spooler worker right way to do this?
>
> I see them die without any failure. There is no way I could find out to
> check if they are working. So I started to think there is something wrong
> with my design.
>
> All I want to do is to run a new listener if someone dies and ensure three
> listeners working. My code is not strictly plain so that I cannot share.
> But if needed, I will try to demonstrate a similar scenario.
>
>
>
> If needed details: I use puka to connect to rabbitmq. Puka has a wait for
> a
> message <http://majek.github.io/puka/puka.html#puka.Client.basic_consume>
> method,
> so I cannot tell a hypothetical monitoring application that "I am alive"
> in
> a timely manner. Because, it does not respond until a message arrives.
> Which I am not sure how much time it will take. Next message may arrive in
> a minute or in a month, or not at all.
>
> Appreciate ideas.
>
>
If understand correctly you need 3 processes that wait on the puka queue,
right ?
If yes, mules are the thing you are searching for.
--
Roberto De Ioris
http://unbit.it
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
