Problems using celery and pyelasticsearch
Hi,
I create a task that uses this function
def get_jobs(specialization, yoe):
try:
key = "SpecAlert-"
if len(specialization):
key += '-'.join(map(str, specialization))
if yoe is not None:
key += '-yoe'.join(yoe)
jobs = memcache_client.get(key)
if not jobs:
spec_param = {'terms': {'jobs.specialization.id':
specialization}}
if yoe is not None:
if len(yoe) == 2:
yoe_param = {'range': {'jobs.experience': {'gte':
yoe[0], 'lte': yoe[1]}}}
elif int(yoe[0]):
yoe_param = {'range': {'jobs.experience': {'gte':
yoe[0]}}}
else:
yoe_param = {'term': {'jobs.experience': yoe[0]}}
query_bool = {'must': [{'range': {'jobs.deadline': {'gt':
str(date.today() + timedelta(days=1))}}}]}
query_bool['must_not'] = [{'term': {'jobs.job_level':
JOB_LEVEL_VOC}}]
if specialization:
query_bool['must'].append(spec_param)
if yoe:
query_bool['must'].append(yoe_param)
es = config.get_es_connection()
es_config = config.config['elasticsearch']
# print({'query': {'bool': query_bool}})
try:
# Tasks sometimes hang here
result = es.search(index=es_config['job_index'],
doc_type=es_config['job_type'],
body={'query': {'bool': query_bool}})
jobs = []
for j in result['hits']['hits']:
jobs.append(j['_source'])
except ElasticsearchException as esc:
print(esc)
jobs = []
if jobs:
memcache_client.set(key, jobs, 3600)
except Exception as e:
jobs = []
print(e)
return jobs
I find that the celery worker often stops executing tasks. After tests
and debugging I in that this NEVER happens when I take out this line(s):
result = es.search(index=es_config['job_index'],
doc_type=es_config['job_type'],
body={'query': {'bool': query_bool}})
This line also does not raise any Exceptions
Does anyone have any idea what could be going on or how I can further
inspect running tasks.
N.B celery worker is started with loglevel=debug flag but does not
output any useful info as regards the problem.
Thanks
--
https://mail.python.org/mailman/listinfo/python-list
Re: Problems using celery and pyelasticsearch
On 07/12/2015 02:50, Laura Creighton wrote: In a message of Mon, 07 Dec 2015 02:37:15 +0100, nonami writes: Does anyone have any idea what could be going on or how I can further inspect running tasks. Not sure this will help, but it might ... https://www.caktusgroup.com/blog/2013/10/30/using-strace-debug-stuck-celery-tasks/ This was really helpful Laura, thanks. -- https://mail.python.org/mailman/listinfo/python-list
Re: About this mailing list
On Fri, Jul 24, 2015 at 9:27 PM, Stéphane Wirtel wrote: > Hi all, > > This mail is just to check if you receive it because I think I have a > problem with this list. > > Could you reply me to check it works ? > We hear you loud and clear > > Thank you > > You are welcome > -- > Stéphane Wirtel - http://wirtel.be - @matrixise > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: How may I learn Python Web Frameworks
The official Django docs is pretty detailed https://docs.djangoproject.com/en/1.8/ You could also look at the Django book but it confesses to being written for version 1.4 even though it goes ahead to assure us that it's not outdated https://django-book.readthedocs.org/en/latest/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Who uses IDLE -- please answer if you ever do, know, or teach
> 0. Classes where Idle is used: > Where? > Level? > > Mostly on windows, can't remember ever using Idle on a linux system before. > Idle users: > > 1. Are you > grade school (1=12)? > undergraduate (Freshman-Senior)? > post-graduate (from whatever)? > Post-graduate > > 2. Are you > beginner (1st class, maybe 2nd depending on intensity of first)? > post-beginner? > Beginner > > 3. With respect to programming, are you > amateur (unpaid) > professional (paid for programming) > > Amateur, Occasionally use python for paid programming but use pycharm, eclipse with PyDev, Or sublime text in those cases mostly. Never Idle. > -- > Terry Jan Reedy, Idle maintainer > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Dave Angel RIP
Folks, Many of us will remember the long-time list regular Dave Angel. Dave was active on this list for many years, a keen participant in many discussions, and very helpful to Python users of all levels of expertise. Unfortunately, I am saddened to say that Dave passed away on May 25th, and the far too young an age of 66. He will be missed. He will be dearly missed. His contributions have been very helpful and will be even more valued now that they are all we have left to remind us of him here on the list. -- https://mail.python.org/mailman/listinfo/python-list
