I've gotten a bit lost in these requirements but having written a large piece of open source data analysis software (including image processing and tiling of data to form and image) there are certain scenarios when processing an image that you can share the entire image with *all* of the threads because each thread works on its own small part of the image and does NOT care about any other region of the image. This is the "embarrassingly parallel" algorithm you always hear about.
Like Konstantin suggested get the total number of "CPUs" that you want to use. Tell each thread what area it will work on and send the pointer to the image to process to each thread. You can work out the user feedback as the threads go through each section. We do this all the time in our program (also built with Qt5). Just my 2 cents. Also an alternate suggestion would be to take a look at Threading Building Blocks (TBB). It has worked out very nicely for us and is a compliment to QThread in certain scenarios. We use both QThread and TBB in our application. -- Mike Jackson https://www.github.com/bluequartzsoftware/DREAM3D On 2/1/22, 10:21 AM, "Interest on behalf of Murphy, Sean" <interest-boun...@qt-project.org on behalf of sean.mur...@centauricorp.com> wrote: > On Mon, Jan 31, 2022 at 7:15 PM Murphy, Sean <mailto:sean.mur...@centauricorp.com> wrote: > > 1. Creating 60,000 QObjects in a single thread appears to be slow > [...] > > Ehm, maybe I'm not understanding something, but why do you need objects to begin with? I do need to report progress back to the UI and I mistakenly thought I needed to have each tile inherit from QObject to provide that functionality. After reading up a bit more about QFuture and QFutureWatcher and then refactoring yesterday to use those classes, as of the moment, my tile class no longer inherits from anything, and I'm able to use signals from QFutureWatcher to relay progress back to the UI. > The actual loop is this: > // generate each tile with its assignment > [snip] > > What's the significance of the tiles? As far as I can tell from your requirements, you don't care about > the "true geometry" of the data. Either I'm understanding what you mean by "true geometry", or this assumption is at least partially incorrect. Looking back on my list of requirements I've posted, I left off the last step: At the end of all this processing, I do need to produce an onscreen image to the user. So any way I slice up the work that needs to be done using threads, once they are all finished, I do need to know what chunk of the original image each thread was working on to know where place its normalized pixels in what I display to the user. > At least to me it seems you want something like (pseudo algorithm): > > 1) Start QThread::idealThreadCount threads (QThread::create<> / std::thread) > 2) Each thread works on "total samples" / QThread::idealThreadCount buffers that are completely independent. > 2.1) Each thread goes through each sample from a partially mapped (from the file) buffer, takes the min/max to get the dynamic range > 2.2) Sync the threads to get the global min/max > 2.3) Go through each of the buffers a second time to normalize the dynamic range (again no tiles involved, just samples) > 3) Done. I think this is an preferable approach to what I was attempting, and I'm glad you suggested it. This being my first attempt at this, I naively started from asking "what seems like it would be a reasonable tile size?", arbitrarily thought "256 pixels square" and worked backwards from there, which is how I got into this mindset of "I might have 60,000+ tiles to deal with". Your approach starts from what now appears to me a much better thought of "what's the ideal number of threads for your machine? Don't bother creating more threads than that because you're not going to benefit by having more" and then working forward from there. > Note: As each thread works on its own piece of data in both cases there's no sync required at any one point - > you just read/write different parts of the same thing. Which is true both for when you load/write from/to a file > and from/to memory. Not sure if I quite understand what you meant by this note? There is the sync you pointed out as your step 2.2, and then since I need to form the results into an onscreen image (most likely a QGraphicsPixmapItem, etc.) there's another sync at your step 3 before I can make the final onscreen image. Otherwise I think I understand and prefer your concept to what I was doing. Thanks again for your help! Now to test this out in practice... Sean _______________________________________________ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest _______________________________________________ Interest mailing list Interest@qt-project.org https://lists.qt-project.org/listinfo/interest
