On Tue, 30 Sep 2008, Nifty niftyompi Mitch wrote:
Also while focusing on network/transport in this discussion none of us made a comment on rotational latency as a source of uncertainty for the kernel state. If we had the ability to synchronize the systems exactly starting a process would lag for want of rotational/seek disk latency in beowulf'y.
Sure, but starting processes is presumed to occur once and then (an efficient computation) proceeds for a long time. All the parallelized tasks effectively arrive at the same point at the computation at the first barrier AFTER all this is finished. If the communications involve giving the kernel a slice at (say) the end when the barrier is released for all nodes at "the same time", then the distributed kernels CAN have roughly the same state IF one suppresses enough of the "random" sources of state-noise -- asynchronous demands for the kernel's attention. To the extent that the kernel can accumulate all of its regular housekeeping and do it an elective time, if one gets the system clocks together within a usec and the kernel does its elective work on clock ticks, that work will end up being done (mostly) synchronously across the nodes. Truthfully, all one is trying to do is to generalize your parallel process to have a double synchronous barrier, with one phase of the computation being "kernel housekeeping". Compute (in parallel) -> barrier (IPCs) -> Kernel (in parallel) -> barrier -> Compute -> barrier -> Kernel -> barrier ... ad inifinitum. If the work done by the kernel is fairly tightly bounded -- predictably completes in (say) 100 usec (which is a LOT of time, far more than one almost ever sees one STILL has 900 usec per tick to work on your compute task. If the kernel (more reasonably) completes in 1-10 usec your cluster should have a 99+% duty cycle but avoid the "noise" that desynchronizes everything.
Shared memory machines and transports will behave differently. The very high accuracy and high precision clock synchronization is a very real problem for some data gathering systems. Once the data is gathered the computation should be less sensitive. These are different problems and might be addressed by the data sampling devices. Synchronization brings problems.... for example a well synchronized campus can hammer yp server and file servers when cron triggers the same actions on 5000+ systems... I try never to fetchmail at the hour, half hour... I suspect that some system cron tasks should no longer run from cron. Common housekeeping tasks necessary for system health should be run via the batch system in a way that is fashionably late enough to not hammer site services.
Absolutely. In fact, you'd want the nodes to be isolated and not running ANY of this stuff, I'd guess. You'd want the nodes to have quiescent, non-demanding hardware (except for devices doing the bidding of the running parallel process) so that nothing "random" needed to be done that couldn't be saved for the kernel slices.
One site service of interest is AC power. A modern processor sitting in an idle state that then starts a well optimized loop will jump from a couple of watts to 100 watts in as many clocks as the set of pipelines is deep behind the instruction decode and instruction cache fill. A 1000 processor (4000 cores) might jump from 4000 watts to 100000 watts in the blink of an eye (err did the lights blink). Buffer that dI/dT through the PS and it is less but still interesting on the mains which are synchronized.
Interesting. I never have seen the lights blink although I don't run synchronous computations. One wonders if the power supply capacitors (which should be quite large, I would think) don't soak up the transient, though, even on very large clusters. Also, I think that the power differential is smaller than you are allowing for -- I don't think most idle processors draw "no" power... rgb
-- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
