weakref, memory management and execution slow down in PyQt4
I built a small application using PyQt4 and pyqtgraph to visualize some data. The app has 32 graphs that plot deques of size 512. The plots are updated when 200 ints are cycled through each deque. The plotting slows down in a linear manner with respect to time. In other words after cycling through 100,000 data points the time between calls to process events is much longer than it was at T0. I have done a little memory profiling. Watching the process on top, it's clear that there is some memory leak. I also tried invoking objgraph.show_most_common_types(). This test reveals that the number of objects being created plateaus, except for weakref objects, which keep growing and growing. I have come to believe that the growing number of weakrefs is slowing down execution. Is my analysis misguided? How can I introspect further? If the slowdown can be attributed to weakref escalation, what are some next steps? Thanks, Kevin 0x8A61431E.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: weakref, memory management and execution slow down in PyQt4
Antoine Pitrou: > kjs riseup.net> writes: >> >> I have come to believe that the growing number of weakrefs is slowing >> down execution. Is my analysis misguided? How can I introspect further? >> If the slowdown can be attributed to weakref escalation, what are some >> next steps? > > The way to analyze this is to build some gradually smaller subsets of your > application until you can isolate what is causing the growth in number of > objects (if any). I would suggest first remove the GUI and replace it with > some dummy functions, to stress your core logic. Thanks for the advice. I commented out the graph generation and PyQt call >>> self.app.processEvents() where in the class __init__ >>> self.app = QtGui.QApplication(sys.argv) This stopped the weakref proliferation. All other objects grow and shrink in number as expected. > > Note that "top" isn't a very reliable tool, as memory fragmentation and > other factors can cause your process' visible size to grow even though > Python's memory consumption may be stable. There are dedicated Python tools > for finer analysis, such as tracemalloc, which is standard on 3.4 and > available > as a backport for older versions: > > https://docs.python.org/3/library/tracemalloc.html > http://pytracemalloc.readthedocs.org/ > > But regardless of such tools, the approach above (try to decompose your > workload into separate parts until your find the culprit) is highly > recommended. > > Regards > > Antoine. > > 0x8A61431E.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: weakref, memory management and execution slow down in PyQt4
Michael Torrie: > On 09/07/2014 01:11 PM, kjs wrote: >> Thanks for the advice. I commented out the graph generation and PyQt call >> >>>>> self.app.processEvents() >> >> where in the class __init__ >> >>>>> self.app = QtGui.QApplication(sys.argv) >> >> This stopped the weakref proliferation. All other objects grow and >> shrink in number as expected. > > Can you make an absolute minimal example with no widgets save the graph > itself? You could then go to the PyQt folks with that. Could be a bug, > or could be an improper use of something by your code. This is one of > the problems with using Qt in other languages: it's not a perfect fit > for Python's object model as it is C++-based. So you have to think like > a C++ programmer, and you might have to manage some resources manually, > unlike normal Python objects which for the most part take care of > themselves. > The code is minimal[0]. The only other widgets are a start button that fires off the plotting and a stop button that calls sys.exit(). Lines 112-114 appear to be causing the weakref proliferation. I do not doubt that I could be using PyQt4 incorrectly. I'll send a message to the pyqt mailing list as well. > Also you could try a minimal example using PySide instead of PyQt and > see if that also manifests the leak. If so, then the problem could be > in Qt itself. Though I suspect the problem is some kind of impedance > mismatch between C++ and Python. > Thanks much, Kevin [0] https://github.com/kevinjos/kaggle-aes-seizure-prediction/blob/master/explore.py 0x8A61431E.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: weakref, memory management and execution slow down in PyQt4
Thanks for the consideration Michael. If you do get the data, and are able to run the code, let me know if you notice anything interesting. Michael Torrie: > On 09/07/2014 02:39 PM, kjs wrote: >> The code is minimal[0]. The only other widgets are a start button that >> fires off the plotting and a stop button that calls sys.exit(). > > Unfortunately there are no data files in your git repository so I can't > run it. The data is available from the internet[0] in the form of 3+GB gziped blobs. In case you don't want to register an account with Kaggle and download 3GB to execute the code, I have uploaded a sample file to a Tahoe LAFS grid accessible via tor. If you're interested in downloading the data, please let me know and I'll share the read capability with you. Additionally, I should write some tests the use mock data. I'll let you know if I get around to this. > >> >> Lines 112-114 appear to be causing the weakref proliferation. > > Is there a reason you are using setattr and getattr instead of a proper > data structure? both of those calls are rather expensive. Would > probably be cheaper to use some kind of array, dictionary, or other > purpose-built data structure? > You're right, a dictionary can do everything I need and more. This happened to be the first thing I thought of, and I didn't imagine it would be very expensive. I figured it was simply a different way of defining and retrieving a class variable. IE setattr(self, foo, True) == self.foo = True. Thanks, Kevin [0] http://www.kaggle.com/c/seizure-prediction/data 0x8A61431E.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: weakref, memory management and execution slow down in PyQt4
On September 9, 2014 8:57:02 AM PDT, Michael Torrie wrote: >On 09/09/2014 09:37 AM, Chris Angelico wrote: >> On Wed, Sep 10, 2014 at 1:32 AM, Michael Torrie >wrote: >>> Yes you're correct. It is the equivalent. But it always involves >>> lookup in the object's dictionary, which is big O order O(n log n) >>> complexity for each and every access. >> >> Where do you get that figure from? A CPython dictionary is >implemented >> as a hashtable, so algorithmic complexity of lookups ought to be >O(1). > >You're right, it is. My mistake. Guess the main issue is the overhead >of an additional function call. A dict directly or a list directly may >me faster. > >Anyway, I guess we're wandering in the weeds a bit as the original >issue >is probably not related to his unorthodox choice of metaprogramming as >a >data model. I agree. I believe I have isolated the issue to be either my user of pyqtgraph, or is implementation. I lean towards the my use, because I know how little effort I put into understanding the library. If I figure on a solution, I'll be sure to come back in. I'm also curious why the weakrefs are not being garbage collected. And how many (~20,000) tiny little things can bring processing to a halt after only a couple minutes of runtime. -- https://mail.python.org/mailman/listinfo/python-list
Re: CSV methodology
[email protected] wrote: > > Hello. Back in the '80s, I wrote a fractal generator, which, over the years, > I've modified/etc to run under Windows. I've been an Assembly Language > programmer for decades. Recently, I decided to learn a new language, > and decided on Python, and I just love it, and the various IDEs. > > Anyway, something I thought would be interesting, would be to export > some data from my fractal program (I call it MXP), and write something > in Python and its various scientific data analysis and plotting modules, > and... well, see what's in there. > > An example of the data: > 1.850358651774470E-0002 > 32 > 22 > 27 > ... (this format repeats) > > So, I wrote a procedure in MXP which converts "the data" and exports > a csv file. So far, here's what I've started with: > > --- > import csv > > fname = 'E:/Users/jayte/Documents/Python Scripts/XportTestBlock.csv' > > f = open(fname) > > reader = csv.reader(f) > > for flt in reader: > x = len(flt) > file.close(f) > --- The csv.reader(f) object creates an iterable that will create lists from lines in f. The list will have values at indexes based on the commas in the file. EX: my_header_1, my_header_2 111, 0001 101, 1010 100, 1001 The csv.reader will lazily make lists like ['my_header_1', 'my_header_2'], ['111', '0001'], ... and so forth. Your program above will take the length of those lists, and assign that value to x. For every line in the file, f will get rewritten with a new value, the length of list which is derived from the number of commas in the csv. Also note that the csv.reader speaks many dialects, and can do similar work on files with different quote characters and delimiters. Generally, I prefer to read in csv files with the standard readline() method of open files. I do like to use csv.DictWriter, which helps me to keep my csv output tabular. -Kevin > > This will get me an addressable array, as: > > flt[0], flt[1], flt[350], etc... from which values can be assigned to > other variables, converted... > > My question: Is there a better way? Do I need to learn more about > how csv file are organized? Perhaps I know far too little of Python > to be attempting something like this, just yet. > > Advice? > > Jeff > 0x8A61431E.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
