"Jim Henderson" <[EMAIL PROTECTED]> posted [EMAIL PROTECTED], excerpted below, on Thu, 10 May 2007 17:23:18 -0600:
> Seems to be related to the number of headers in the group - I had > forgotten to set the purge for the server, so I had messages going back > several years. > > Unlike your issue, I'm not seeing the processor utilization spike - it > could just be the raw number of headers having to be sorted each time > that was throwing it off. > > But I'll poke through the open bugs and see if any of the issues I've > run into are in there - I'd forgotten that Charles had set this up to > track bugs. > > Jim > > On 5/10/07, SciFi <[EMAIL PROTECTED]> wrote: >> On Thu, 10 May 2007 19:05:38 +0000, Jim Henderson wrote: >> > >> > One other thing that I had forgotten to mention - the download >> > performance is AWESOME (as I knew it was in these newer builds), but >> > I've had points where Pan totally seizes (it eventually does come >> > back to life, but while it's stuck it's seemingly hung) when >> > downloading groups or switching into a group that has many tens of >> > thousands of headers in it. >> >> I opened this bug: >> <http://bugzilla.gnome.org/show_bug.cgi?id=430628> There are a few >> others depending on exact scenario. Please check bugzilla and chime-in >> there on which ones may be affecting you. It's a sure way to "vote" >> for which bugs are, ummm, bugging most ppl. ;) >> >> > [...] Please don't top post. It messes up quote context. There's a reason pan warns about it. Please respect that on the pan list, even if you top post elsewhere. Old-pan was fully multithreaded. This increased performance some and responsiveness dramatically, but made coding and debugging timing issues between the threads incredibly difficult. In the rewrite, Charles decided that the added complexity wasn't worth it in the main, so while pan now spins off worker threads in some cases (when first negotiating connections, consolidating back to a single thread after they are established, and either just added or actively being tested, for decoding), the main loop, group updating, and the GUI are all single threaded. What's happening is that pan is only multi-threaded now where a single thread creates a noticeable bottleneck. The first place that was noticed was at initial connection, where it would sometimes take 10-20 minutes to establish all the available connections and get up to full speed. If one was only downloading a moderate amount, and at full speed could have accomplished the entire download in 10-20 minutes, this was obviously very frustrating as it could increase the time to completion by 50-100%. I was one of the ones that complained about that. So Charles made connection establishment multithreaded, and that bottleneck went away. Then someone that was saving attachments directly noticed that pan would idle the connections sometimes while it decoded and saved big attachments (consisting of 50 yencoded parts or so). Since it's generally the download that takes the time, seeing it sit there idle, increasing the total time to completion just to do the relatively fast local decode and save was very frustrating. (I didn't see this one, as I tend to work in multiple steps, downloading to cache while I do other things, then coming back when everything's in local cache to actually sort thru things and decide what I want to keep and where, and what I want to kill. Thus the decode and save wasn't occurring at the same time as the actual downloading, and couldn't slow the downloading down.) As for this new issue, bottlenecking on loading a group... * If you have download new headers when entering group set, you'll see more of a slowdown. Try turning that off and triggering your header downloads only manually. That should help. * pan now message-threads dynamically as it gets headers, saving the threaded list to disk. This takes FAR less memory as various tricks are used to combine similar strings and on multisegment posts, display only one header for however many individual segments make up the part. This is more work (and accomplished in that single thread) when downloading headers, but where old-pan would do it every time it started (thus the minutes long load time if you had many headers to thread), new-pan does it once. Of course, loading the pre-threaded headers still takes some CPU and perhaps more importantly disk i/o time. Since pan now doesn't keep everything in memory at once as it used to do, only a single group, it reloads this pre-threaded list every time you enter the group. * The follow-on from the above is that at least if you have enough memory to give the system cache to work with in addition to the loaded group, you can decrease the stalls further by loading each group before you start your download, then starting your download in one before reloading the other group. pan will still load the pre-threaded message list when you reload the group, but it'll already be in cache, so you won't have the disk activity to worry about. * As mentioned above, I'm not been following the decode-and-save multi- threading very closely, as I don't have the issue the way I do things, but I'm not sure if it has been merged into trunk or is still being tested outside of trunk. If it hasn't been merged to trunk yet, you'll see slowdowns during decode and save, and thus be able to avoid them by using my technique of downloading to cache first (while working on something else, or sleeping/working/shopping/whatever), then coming back when everything's in local cache to do your actual sorting and saving of attachments to disk. You'll need to increase your cache size, but I cover that in other posts. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman _______________________________________________ Pan-users mailing list Pan-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/pan-users
