Cancel threads after timeout
Here's the use case I want to implement - I have to generate a report from multiple database servers. This report should be generated every 2 hours. Sometimes it happens that a query on one of the database servers takes longer than expected and impedes the generation of this report (that's right, the queries are ran sequential). What I am trying to achieve is to parallelize the queries on each database server and to be able to cancel one of them if it takes longer than X minutes. threading.Thread doesn't support this and seems that in general programming languages don't implement a way to cancel threads from the outside. Now, I've read all the stackoverflow threads about killing a thread, canceling a thread after a timeout, but all of them imply that you are able to check from within the thread if you should end the computation or not - that's not really my case, where the computation is a SQL query. So, what I have in mind is something like: the main loop starts a threading.Thread which in turn is responsible for starting another thread in which the actual computation happens (could be a threading.Thread or a multiprocessing.Process) *and* checks if the specified timeout has passed. If the time is up, it exits, letting the main loop know. Lots of words, no code - let me know if you have any suggestions, ideas to this rant. Thanks! -- http://mail.python.org/mailman/listinfo/python-list
Re: Cancel threads after timeout
On 01/26, Matt Jones wrote: The SQL part is not under control and getting it there requires more development time than the management is willing to allocate. So this would be a first step before refactoring that part. I'm aware that the database might not notice that I don't want to receive the result of the query, but I'm willing to assume that risk for now. Thank you. > It sounds like your real problem is with your SQL query... Is that part of > this problem under your control? Can you break the query into smaller, > quicker, pieces that you can run in a reasonable amount of time? > > If not, nesting threads might be your best programmatic solution. Heed > Jason's warning though that the SQL Server my still be working even if you > cancel an operation from the outside (which could compound your problem). > > *Matt Jones* > > > On Sat, Jan 26, 2013 at 9:43 AM, Jason Friedman wrote: > > > > Sometimes it happens that a query on one of the database servers > > > takes longer than expected and impedes the generation of this report > > > (that's right, the queries are ran sequential). What I am trying to > > > achieve is to parallelize the queries on each database server and to be > > > able to cancel one of them if it takes longer than X minutes. > > > > Only answering a small portion of your question > > Assuming you are able to figure out how to "cancel" a thread or > > process on your side, it is possible the database itself will not > > respect that. In other words, if you execute "SELECT ..." singly, > > outside of Python, and type CNTL-C, does your database quickly > > recognize you are no longer interested in the result set and stop its > > work? > > -- > > http://mail.python.org/mailman/listinfo/python-list > > > -- > http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
python interfaces
Hi, Probably it has been asked before, but I'll still ask. Why doesn't python provide interfaces trough its standard library? Or it was ever proposed to be included in the language? Zope's implementation seems pretty flexible and straightforward. Thanks. -- http://mail.python.org/mailman/listinfo/python-list
advanced usage of python threads
Hi, Is there a document where I can find some advanced information about python threads? I know the basic things about them and did some practice, but when I try to advance I don't know where to go or how to go. Thanks. -- http://mail.python.org/mailman/listinfo/python-list
Re: advanced usage of python threads
Well, I will be writing the application server of a three-tier architecture system. I will be using Twisted for the communication with the client but from there I have to make several calls to a database and this asks threading. The tables will be filled by another system that gathers some data. I want to implement several threads (workers) that takes care of each client and a coordinator thread that takes care of all the threads. But I have never worked with threads at this level and I feel like I need some coordination. I will be thankful with documentation on advanced threads, I probably can apply that in practice. Thanks to all. -- http://mail.python.org/mailman/listinfo/python-list
Re: advanced usage of python threads
Chris, I have already made my choice, I am asking just for a little help with some documentation. I know about twisted.enterprise.adbapi, but the company is working with sqlalchemy at the time. So please, I know you have good intentions but you're kind of not helping me :) Well, probably instead of asking you people I should just start searching the net, thing that I am already doing. But if any of you has some info, please share. Thanks. -- http://mail.python.org/mailman/listinfo/python-list
decorators don't play nice with nose?
Hi, I am trying to test the business part of a web service. For this I
am using unittest & nose.
I wrote a decorator that should handle the xml test file retrieval, but
it seems I can't get it working with nose.
Here's the code:
* MyApp.py -- base test class *
import os
import unittest
from MyApp.Core import XmlParser
__all__ = ['MyAppTest', 'setup']
PATH = os.path.dirname(__file__) or ''
class setup(object):
"""Decorator to ease the use of xml files in MyApp tests.
The way it works it that it decorates a test method which has a first
default parameter called 'parser' and it overwrites this parameter value
with a XmlParser instance.
The xml file should be located under:
data/testedBusinessRequest/testMethodName.xml
"""
def __init__(self, testedBusinessRequest = ''):
self.testedBusinessRequest =\
testedBusinessRequest.lower()
def _getXmlParser(self, xml):
documentElement = XmlParser.parseXmlStream(xml)
parser = XmlParser.getParser(documentElement)
return parser
def __call__(self, method):
# TODO: error handling here
methodName = method.func_code.co_name
methodName = methodName.split('_')[1]
xmlFolder = self.testedBusinessRequest
xmlFile = '%s.xml' % methodName
path = os.path.join(PATH, 'data',
xmlFolder, xmlFile)
f = open(path)
xml = f.read()
f.close()
method.func_defaults = (self._getXmlParser(xml),)
return method
class MyAppTest(unittest.TestCase):
def setUp(self):
self.database = Database()
def tearDown(self):
pass
* test_Login.py - test a business request *
from MyAppTest import MyAppTest, setup
from MyApp import Login
class TestLogin(MyAppTest):
testedBusinessRequest = 'Login'
@setup(testedBusinessRequest)
def test_validParameters(self, parser = None):
response = Login(self.database, parser).run()
return True
Ok, so the decorator setup should fill the parser parameter with a
XmlParser object. This works well if I add a __main__ and use unittest
to run the tests. But if I use nose, I get the following error:
*TypeError: unbound method __call__() must be called with setup instance
as first argument (got module instance instead)*
Any advices?
Thanks!
--
http://mail.python.org/mailman/listinfo/python-list
scalable xml
Hi, I am writing the application server for a three-tier architecture and sending the client's response in xml. My question is: is there a way to build the xml dom in a more scalable way and faster way than just creating every textNode and element for it? I have tons of data to transmit and it drives me crazy having to build that dom manually. I am not sure if this is a stupid question as I don't know other alternatives ... maybe just provide a template xml which I can fill with data but that can introduce some pretty ugly bugs in the application. Thanks. -- http://mail.python.org/mailman/listinfo/python-list
