On 2012-10-31 CJ O'Reilly asked some pertinent questions about HPC, Cluster, Beowulf computing from the perspective of a newbie:
http://www.beowulf.org/pipermail/beowulf/2012-October/030359.html The replies began in the "Digital Image Processing via HPC/Cluster/Beowulf - Basics" thread on the November pages of the Beowulf Mailing List archives. (The archives strip off the "Re: " from the replies' subject line.) http://www.beowulf.org/pipermail/beowulf/2012-November/thread.html Mark Hahn wrote a response which I and MC O'Reilly found most informative. For the benefit of other lost souls wandering into this fascinating field, I nominate Mark's reply as a: Beowulf/Cluster/HPC FAQ for newbies Mark Hahn 2012-11-03 http://www.beowulf.org/pipermail/beowulf/2012-November/030363.html Quite likely the other replies and those yet to come will be well worth reading too, so please check the November thread index, and potentially December if the discussions continue there. Mark wrote a second response exploring options for some hypothetical image processing problems. Googling for an HPC or Cluster FAQ lead to FAQs for people who are users of specific clusters. I couldn't easily find a FAQ suitable for people with general computer experience wondering what they can and can't do, or at least what would be wise to do or not do, with clusters, High Performance Computing etc. The Wikipedia articles: http://en.wikipedia.org/wiki/High-performance_computing http://en.wikipedia.org/wiki/High-throughput_computing are of general interest. Here are some thoughts which might complement what Mark wrote. I am a complete newbie to this field and I hope more experienced people will correct or expand on the following. HPC has quite a long history and my understanding of the Beowulf concept is to make clusters using easily available hardware and software, such as operating systems, servers and interconnects. The Interconnects, I think are invariably Ethernet, since anything else is, or at least was until recently, exotic and therefore expensive. Traditionally - going back a decade or more - I think the assumptions have been that a single server has one or maybe two CPU cores and that the only way to get a single system to do more work is to interconnect a large number of these servers with a good file system and fast inter-server communications so that suitably written software (such as something written to use MPI so the instances of the software on multiple servers and/or CPU cores and work on the one problem efficiently) can do the job well. All these things - power supplies, motherboards, CPUs, RAM, Ethernet (one or two 1GBps Ethernet on the motherboard) and Ethernet switches are inexpensive and easy to throw together into a smallish cluster. However, creating software to use it efficiently can be very tricky - unless of course the need can be satisfied by MPI-based software which someone else has already written. For serious work, the cluster and its software needs to survive power outages, failure of individual servers and memory errors, so ECC memory is a good investment . . . which typically requires more expensive motherboards and CPUs. I understand that the most serious limitation of this approach is the bandwidth and latency (how long it takes for a message to get to the destination server) of 1Gbps Ethernet. The most obvious alternatives are using multiple 1Gbps Ethernet connections per server (but this is complex and only marginally improves bandwidth, while doing little or nothing for latency) or upgrading to Infiniband. As far as I know, Infiniband is exotic and expensive compared to the mass market motherboards etc. from which a Beowulf cluster can be made. In other words, I think Infiniband is required to make a cluster work really well, but it does not not (yet) meet the original Beowulf goal of being inexpensive and commonly available. I think this model of HPC cluster computing remains fundamentally true, but there are two important developments in recent years which either alter the way a cluster would be built or used or which may make the best solution to a computing problem no longer a cluster. These developments are large numbers of CPU cores per server, and the use of GPUs to do massive amounts of computing, in a single inexpensive graphic card - more crunching than was possible in massive clusters a decade earlier. The ideal computing system would have a single CPU core which could run at arbitrarily high frequencies, with low latency, high bandwidth, access to an arbitrarily large amount of RAM, with matching links to hard disks or other non-volatile storage systems, with a good Ethernet link to the rest of the world. While CPU clock frequencies and computing effort per clock frequency have been growing slowly for the last 10 years or so, there has been a continuing increase in the number of CPU cores per CPU device (typically a single chip, but sometimes multiple chips in a device which is plugged into the motherboard) and in the number of CPU devices which can be plugged into a motherboard. Most mass market motherboards are for a single CPU device, but there are a few two and four CPU motherboards for Intel and AMD CPUs. It is possible to get 4 (mass market) 6, 8, 12 or sometimes 16 CPU cores per CPU device. I think the 4 core i7 CPUs or their ECC-compatible Xeon equivalents are marginally faster than those with 6 or 8 cores. In all cases, as far as I know, combining multiple CPU cores and/or multiple CPU devices results in a single computer system, with a single operating system and a single body of memory, with multiple CPU cores all running around in this shared memory. I have no clear idea how each CPU core knows what the other cores have written to the RAM they are using, since each core is reading and writing via its own cache of the memory contents. This raises the question of inter-CPU-core communications, within a single CPU chip, between chips in a multi-chip CPU module, and between multiple CPU modules on the one motherboard. It is my impression that the AMD socket G34 Opterons: http://en.wikipedia.org/wiki/Opteron#Socket_G34 Magny-Cours Opteron 6100 and Interlagos 6200 devices solve these problems better than current Intel chips. For instance, it is possible to buy 2 and 4 socket motherboards (though they are not mass-market, and may require fancy power supplies) and then to plug in 8, 12 or 16 core CPU devices into them, with a bunch of DDR3 memory (ECC or not) and so make yourself a single shared memory computer system with compute power which would have only been achievable with a small cluster 5 years ago. I understand the G34 Opterons have a separate Hypertransport link between any one CPU-module and the other three CPU modules on the motherboard on a 4 CPU-module motherboard. I understand that MPI works identically from the programmer's perspective between CPU-cores on a shared memory computer as between CPU-cores on separate servers. However, the performance (low latency and high bandwidth) of these communications within a single shared memory system is vastly higher than between any separate servers, which would rely on Infiniband or Ethernet. So even if you have, or are going to write, MPI-based software which can run on a cluster, there may be an argument for not building a cluster as such, but for building a single motherboard system with as many as 64 CPU cores. I think the major new big academic cluster projects focus on getting as many CPU cores as possible into a single server, while minimising power consumption per unit of compute power, and then hooking as many as possible of these servers together with Infiniband. Here is a somewhat rambling discussion of my own thoughts regarding clusters and multi-core machines, for my own purposes. My interests in high performance computing involve music synthesis and physics simulation. There is an existing, single-threaded (written in C, can't be made multithreaded in any reasonable manner) music synthesis program called Csound. I want to use this now, but as a language for synthesis, I think it is extremely clunky. So I plan to write my own program - one day . . . When I do, it will be written in C++ and multithreaded, so it will run nicely on multiple CPU-cores in a single machine. Writing and debugging a multithreaded program is more complex than doing so for a single-threaded program, but I think it will be practical and a lot easier than writing and debugging an MPI based program running either on on multiple servers or on multiple CPU-cores on a single server. I want to do some simulation of electromagnetic wave propagation using an existing and widely used MPI-based (C++, open source) program called Meep. This can run as a single thread, if there is enough RAM, or the problem can be split up to run over multiple threads using MPI communication between the threads. If this is done on a single server, then the MPI communication is done really quickly, via shared memory, which is vastly faster than using Ethernet or Inifiniband to other servers. However, this places a limit on the number of CPU-cores and the total memory. When simulating three dimensional models, the RAM and CPU demands can easily become extremely demanding. Meep was written to split the problem into multiple zones, and to work efficiently with MPI. For Csound, my goal is to get a single piece of music synthesised in as few hours as possible, so as to speed up the iterative nature of the cycle: alter the composition, render it, listen to it and alter the composition again. Probably the best solution to this is to buy a bleeding edge clock-speed (3.4GHz?) 4-core Intel i7 CPU and motherboard, which are commodity consumer items for the gaming market. Since Csound is probably limited primarily by integer and floating point processing, rather than access to large amounts of memory or by reading and writing files, I could probably render three or four projects in parallel on a 4-core i7, with each rendering nearly as fast as if just one was being rendered. However, running four projects in parallel is of only marginal benefit to me. If I write my own music synthesis program, it will be in C++ and will be designed for multithreading via multiple CPU-cores in a shared memory (single motherboard) environment. It would be vastly more difficult to write such a program using MPI communications. Music projects can easily consume large amounts of CPU and memory power, but I would rather concentrate on running a hot-shot single motherboard multi-CPU shared memory system and writing multi-threaded C++ than to try to write it for MPI. I think any MPI-based software would be more complex and generally slower, even on a single server (MPI communications via shared memory rather than Ethernet/Infiniband) than one which used multiple threads each communicating via shared memory. Ten or 15 years ago, the only way to get more compute power was to build a cluster and therefore to write the software to use MPI. This was because CPU-devices had a single core (Intel Pentium 3 and 4) and because it was rare to find motherboards which handled multiple such chips. Now, with 4 CPU-module motherboards it is totally different. A starting point would be to get a 2-socket motherboard and plug 8-core Opterons into it. This can be done for less than $1000, not counting RAM and power supply. For instance the Asus KGPE-D16 can be bought for $450. 8 core 2.3GHz G34 Opterons can be found on eBay for $200 each. I suspect that the bleeding edge mass-market 4 core Intel i7 CPUs are probably faster per core than these Opterons. I haven't researched this, but they are a much faster clock-rate at 3.6GHz and the CPU design is the latest product of Intel's formidable R&D. On the other hand, the Opterons have Hypertransport and I think have a generally strong floating point performance. (I haven't found benchmarks which reasonably compare these.) I guess that many more 8 and 12 core Opterons may come on the market as people upgrade their existing systems to use the 16 core versions. The next step would be to get a 4 socket motherboard from Tyan or SuperMicro for $800 or so and populate it with 8, 12 or (if money permits) 16 core CPUs and a bunch of ECC RAM. My forthcoming music synthesis program would run fine with 8 or 16GB of RAM. So one or two of these 16 (2 x 8) to 64 (4 x 16) core Opteron machines would do the trick nicely. This involves no cluster, HPC or fancy interconnect techniques. Meep can very easily be limited by available RAM. A cluster solves this, in principle, since it is easy (in principle) to get an arbitrary number of servers and run a single Meep project on all of them. However, this would be slow compared to running the whole thing on multiple CPU-cores in a single server. I think the 4 socket G43 Opteron motherboards are probably the best way to get a large amount of RAM into a single server. Each CPU-module socket has its own set of DDR3 RAM, and there are four of these. The Tyan S8812: http://www.tyan.com/product_SKU_spec.aspx?ProductType=MB&pid=670&SKU=600000180 has 8 memory slots per CPU-module socket. Populated with 32 x 8GB ECC memory at about $80 each, this would be 256GB of memory for $2,560. As far as I know, if the problem requires more memory than this, then I would need to use multiple servers in a cluster with MPI communications via Ethernet or Infiniband. However, if the problem can be handled by 32 to 64 CPU-cores with 256GB of RAM, then doing it in a single server as described would be much faster and generally less expensive than spreading the problem over multiple servers. The above is a rough explanation for why the increasing number of CPU-cores per motherboard, together with increased number of DIMM slots per motherboard with increased affordable memory per DIMM slot means that many projects which in the past could only be handled with a cluster, can now be handled faster and less expensively with a single server. This "single powerful multi-core server" approach is particularly interesting to me regarding writing my own programs for music synthesis program or physics simulation. The simplest approach would be write the software in single-threaded C++. However, that won't make use of the CPU power, so I need to use the inbuilt multithreading capabilities of C++. This requires a more complex program design, but I figure I can cope with this. In principle I could write a single-thread physics simulation program and access massive memory via a 4 CPU-module Opteron motherboard, but any such program would be performance limited by the CPU speed, so it would make sense to split it up over multiple CPU-cores with multithreading. For my own projects, I think this is the way forward. As far as I can see, I won't need more memory than 256GB and more CPU power than I can get on a single motherboard. If I did, then I would need to write or use MPI-based software and build a cluster - or get some time on an existing cluster. There is another approach as well - writing software to run on GPUs. Modern mass-market graphics cards have dozens or hundreds of CPUs in what I think is a shared, very fast, memory environment. These CPUs are particularly good at floating point and can be programmed to do all sorts of things, but only using a specialised subset of C / C++. I want to write in full C++. However, for many people, these GPU systems are by far the cheapest form of compute power available. This raises questions of programming them, running several such GPU board in a single server, and running clusters of such servers. The recent thread on the Titan Supercomputer exemplifies this approach - get as many CPU-cores and GPU-cores as possible onto a single motherboard, in a reasonably power-efficient manner and then wire as many as possible together with Infiniband to form a cluster. Mass market motherboards and graphics cards with Ethernet is arguably "Beowulf". If and when Infiniband turns up on mass-market motherboards without a huge price premium, that will be "Beowulf" too. I probably won't want or need a cluster, but I lurk here because I find it interesting. Multi-core conventional CPUs (64 bit Intel x86 compatible, running GNU-Linux) and multiple such CPU-modules on a motherboard are chewing away at the bottom end of the cluster field. Likewise GPUs if the problem can be handled by these extraordinary devices. With clusters, since the inter-server communication system (Infiniband or Ethernet) - with the accompanying software requirements (MPI and splitting the problem over multiple servers which do not share a single memory system) is the most serious bottleneck, the best approach seems to be to make each server as powerful as possible, and then use as few of them as necessary. - Robin http://www.firstpr.com.au _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
