On Thu, 2018-04-05 at 16:18 +0000, Akins, Christopher (GE Aviation, Non-GE, US) wrote: > […] > I'm trying to modernize the process, and SCons has come favorably to my > attention. Due to the lengthy and memory-intensive simulations, though, it > would be useful if we could use the same build system to assign tasks to > different nodes in a cluster. I'm willing to sink some time into making > the necessary mods myself - for fault-tolerance, retries, etc - but I > thought I'd ask if this had come up in the past. If it hasn't, would there > be interest in having a distributed feature, or would it be more of a > private fork?
Off the top of my head: I would keep the decision making all on a single master node. SCons is controlled by an acyclic directed graph and whilst there is Hazelcast etc. Python is a single-threaded, single CPU progrmming language at its heart: keeping data structures in a single PVM is the best way forward. Distributing discrete, self-contained tasks which are essentially pure functions makes a great deal of sense for seriously big builds. I did some Python and SCons training for a company in 2007 who had a not dissimilar problem to the one I think is being described here, and it worked very well. Though I suspect Parts would have helped a lot and saved a lot of hand crafting. For distribution, 0MQ and a dedicated application level communications protocol may well help. Traditionally, Python has not been the best language for handling asynchronous activity, Go or Java (or Scala, Kotlin, Groovy, etc) were the "go to" languages since they have threadpools and tasks so as to avoid messing around with single threaded event loops and callbacks. I haven't plaued with the async/await in Python 3.5+ nor asyncio, but it is a single threaded event loop, and I am a blocked task in a thread pool sort of person as "callback hell" is not where I want to go. Now that SCons is a Python 3 system, this stuff should be tried. I wouldn't start anything new in this area until after a conversation with Jason about Parts. -- Russel. ========================================= Dr Russel Winder t:+44 20 7585 2200 41 Buckmaster Road m:+44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk
signature.asc
Description: This is a digitally signed message part
_______________________________________________ Scons-dev mailing list [email protected] https://pairlist2.pair.net/mailman/listinfo/scons-dev
