Prentice: What cpu or cpus, amount of memory and how many hard drives are on a single node. I may have the power draw for that configuration.
Larry -----Original Message----- From: Beowulf [mailto:beowulf-boun...@beowulf.org] On Behalf Of beowulf-requ...@beowulf.org Sent: Monday, July 28, 2014 12:00 PM To: beowulf@beowulf.org Subject: Beowulf Digest, Vol 125, Issue 13 Send Beowulf mailing list submissions to beowulf@beowulf.org To subscribe or unsubscribe via the World Wide Web, visit http://www.beowulf.org/mailman/listinfo/beowulf or, via email, send a message with subject or body 'help' to beowulf-requ...@beowulf.org You can reach the person managing the list at beowulf-ow...@beowulf.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Beowulf digest..." Today's Topics: 1. Power draw of cluster nodes under heavy load (Prentice Bisbal) 2. Re: Power draw of cluster nodes under heavy load (Jeff White) 3. Re: Power draw of cluster nodes under heavy load (Michael Di Domenico) 4. Re: Power draw of cluster nodes under heavy load (Mark Hahn) 5. Re: Power draw of cluster nodes under heavy load (Prentice Bisbal) 6. Re: Power draw of cluster nodes under heavy load (Prentice Bisbal) ---------------------------------------------------------------------- Message: 1 Date: Mon, 28 Jul 2014 10:51:12 -0400 From: Prentice Bisbal <prentice.bis...@rutgers.edu> To: "beowulf@beowulf.org" <beowulf@beowulf.org> Subject: [Beowulf] Power draw of cluster nodes under heavy load Message-ID: <53d66360.1050...@rutgers.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Beowulfers, Are any of you monitoring the power draw on your clusters? If so, can any of you provide me with some statistics on your power draw under heavy load? Ideally, I'm looking for the power load for a worst-case scenario, such as running HPL, on a per-rack basis. If you can provide me with the power draw and a description of the hardware, that would be great. I have some numbers from a friend who lurks on this list, but the more data points I have, the better I can justify my power requirements for a new cluster purchase I'm working on. -- Prentice ------------------------------ Message: 2 Date: Mon, 28 Jul 2014 13:29:35 -0400 From: Jeff White <jaw...@pitt.edu> To: <beowulf@beowulf.org> Subject: Re: [Beowulf] Power draw of cluster nodes under heavy load Message-ID: <53d6887f.60...@pitt.edu> Content-Type: text/plain; format=flowed; charset="ISO-8859-1" Power draw will vary greatly depending on many factors. Where I am at we currently have 16 racks of HPC equipment (compute nodes, storage, network gear, etc.) using about 140kVA but can use up to 160 kVA. A single rack with 26 compute nodes each with 64 cores worth of AMD 6276 (Supermicro boxes) is using about 18 kW across the PDUs, 3 phase at 240 volts, with most of the nodes at 100% CPU usage. Jeff White - GNU+Linux Systems Administrator University of Pittsburgh - CSSD On 07/28/2014 10:51 AM, Prentice Bisbal wrote: > Beowulfers, > > Are any of you monitoring the power draw on your clusters? If so, can > any of you provide me with some statistics on your power draw under > heavy load? Ideally, I'm looking for the power load for a worst-case > scenario, such as running HPL, on a per-rack basis. If you can provide > me with the power draw and a description of the hardware, that would be > great. > > I have some numbers from a friend who lurks on this list, but the more > data points I have, the better I can justify my power requirements for a > new cluster purchase I'm working on. > ------------------------------ Message: 3 Date: Mon, 28 Jul 2014 13:38:21 -0400 From: Michael Di Domenico <mdidomeni...@gmail.com> Cc: Beowulf Mailing List <beowulf@beowulf.org> Subject: Re: [Beowulf] Power draw of cluster nodes under heavy load Message-ID: <CABOsP2O7qNNHxC0nvUs=s22bkqfvbnfgrhahmht_ekp87oy...@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 I can't go into specifics. with HPL i usually can't top 80-85% of full power draw per cabinet. we have cabinets ranging from 17kw, 25kw, and 35kw. some of our user codes can push the machines to 90% of full load. this is shown on both amd 4-socket nodes and intel dual-socket with gpu's. we're running 208v three phase to the racks and 208v single phase to each server On Mon, Jul 28, 2014 at 1:29 PM, Jeff White <jaw...@pitt.edu> wrote: > Power draw will vary greatly depending on many factors. Where I am at we > currently have 16 racks of HPC equipment (compute nodes, storage, network > gear, etc.) using about 140kVA but can use up to 160 kVA. A single rack > with 26 compute nodes each with 64 cores worth of AMD 6276 (Supermicro > boxes) is using about 18 kW across the PDUs, 3 phase at 240 volts, with most > of the nodes at 100% CPU usage. > > Jeff White - GNU+Linux Systems Administrator > University of Pittsburgh - CSSD > > > On 07/28/2014 10:51 AM, Prentice Bisbal wrote: >> >> Beowulfers, >> >> Are any of you monitoring the power draw on your clusters? If so, can >> any of you provide me with some statistics on your power draw under >> heavy load? Ideally, I'm looking for the power load for a worst-case >> scenario, such as running HPL, on a per-rack basis. If you can provide >> me with the power draw and a description of the hardware, that would be >> great. >> >> I have some numbers from a friend who lurks on this list, but the more >> data points I have, the better I can justify my power requirements for a >> new cluster purchase I'm working on. >> > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf ------------------------------ Message: 4 Date: Mon, 28 Jul 2014 14:13:48 -0400 (EDT) From: Mark Hahn <h...@mcmaster.ca> To: Prentice Bisbal <prentice.bis...@rutgers.edu> Cc: "beowulf@beowulf.org" <beowulf@beowulf.org> Subject: Re: [Beowulf] Power draw of cluster nodes under heavy load Message-ID: <alpine.lfd.2.02.1407281220230.30...@coffee.psychology.mcmaster.ca> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed > Are any of you monitoring the power draw on your clusters? If so, can any of > you provide me with some statistics on your power draw under heavy load? good question; it's something that deserves more attention and coverage. ATM, I can only provide one non-answer: http://www.sharcnet.ca/~hahn/saw-power-by-node.png this is active mixed-user load (45 unrelated users, approximately 85% CPU utilization due to memory scheduling and job layout constraints). this an older cluster, HP dual-socket E5440 (2.833G) whose IPMI happens to return nice power measures. > Ideally, I'm looking for the power load for a worst-case scenario, such as > running HPL, on a per-rack basis. I don't understand the "per-rack" part - aren't you interested in per-node? > I have some numbers from a friend who lurks on this list, but the more data > points I have, the better I can justify my power requirements for a new > cluster purchase I'm working on. my experience is that vendors are useless in this regard: they always want to quote the PSU max rating, and then often don't even use the number right. (ie, put all the low-dissipation stuff like networking together, etc.) has anyone tried to rate the accuracy of vendor power calculators? at least a few years ago, they were absurdly inflated. regards, mark hahn. ------------------------------ Message: 5 Date: Mon, 28 Jul 2014 14:53:05 -0400 From: Prentice Bisbal <prentice.bis...@rutgers.edu> To: Mark Hahn <h...@mcmaster.ca> Cc: "beowulf@beowulf.org" <beowulf@beowulf.org> Subject: Re: [Beowulf] Power draw of cluster nodes under heavy load Message-ID: <53d69c11.6000...@rutgers.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 07/28/2014 02:13 PM, Mark Hahn wrote: >> Are any of you monitoring the power draw on your clusters? If so, can >> any of you provide me with some statistics on your power draw under >> heavy load? > > good question; it's something that deserves more attention and coverage. > > ATM, I can only provide one non-answer: > > http://www.sharcnet.ca/~hahn/saw-power-by-node.png > > this is active mixed-user load (45 unrelated users, approximately 85% > CPU utilization due to memory scheduling and job layout constraints). > this an older cluster, HP dual-socket E5440 (2.833G) whose IPMI > happens to > return nice power measures. Thanks. That image is more helpful than you think - I didn't even think of using IPMI to report power consumption. Using that, I could run HPL on some nodes here and get measurements. > > >> Ideally, I'm looking for the power load for a worst-case scenario, >> such as running HPL, on a per-rack basis. > > I don't understand the "per-rack" part - aren't you interested in > per-node? Ideally, per-node is even better, but I figured most measurements would be at the PDU or circuit level, with one or two PDUs/Circuits per rack. I figured this is the granularity most people are measuring at, which is why I asked that way. > > >> I have some numbers from a friend who lurks on this list, but the >> more data points I have, the better I can justify my power >> requirements for a new cluster purchase I'm working on. > > my experience is that vendors are useless in this regard: they always > want > to quote the PSU max rating, and then often don't even use the number > right. > (ie, put all the low-dissipation stuff like networking together, etc.) > > has anyone tried to rate the accuracy of vendor power calculators? > at least a few years ago, they were absurdly inflated. This is why I'm asking for actual, measured numbers. I read a whitepaper by APC or Raritan that said that if you go with the nameplate on a PDU, you can oversize your power requirements by a factor of 2x. For HPC, I imagine it wouldn't be that extreme, since cluster nodes tend to be at 100% more of the time and therefore use more power. One vendor said they assume 60% - 90% of nameplate ratings when estimating power needs, which is still a pretty broad range. > > regards, mark hahn. ------------------------------ Message: 6 Date: Mon, 28 Jul 2014 14:55:38 -0400 From: Prentice Bisbal <prentice.bis...@rutgers.edu> To: beowulf@beowulf.org Subject: Re: [Beowulf] Power draw of cluster nodes under heavy load Message-ID: <53d69caa.20...@rutgers.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 07/28/2014 01:29 PM, Jeff White wrote: > Power draw will vary greatly depending on many factors. Where I am at > we currently have 16 racks of HPC equipment (compute nodes, storage, > network gear, etc.) using about 140kVA but can use up to 160 kVA. A > single rack with 26 compute nodes each with 64 cores worth of AMD 6276 > (Supermicro boxes) is using about 18 kW across the PDUs, 3 phase at > 240 volts, with most of the nodes at 100% CPU usage. Agreed there's a lot of variability. Since I don't exactly what's going in my new space yet, I'm looking for everyone's input to come up with an average, or ballpark amount. the 5 - 10 kW one vendor specified seems waaaay too low for a rack of high-density HPC nodes running at or near 100% utilization. > > Jeff White - GNU+Linux Systems Administrator > University of Pittsburgh - CSSD > > On 07/28/2014 10:51 AM, Prentice Bisbal wrote: >> Beowulfers, >> >> Are any of you monitoring the power draw on your clusters? If so, can >> any of you provide me with some statistics on your power draw under >> heavy load? Ideally, I'm looking for the power load for a worst-case >> scenario, such as running HPL, on a per-rack basis. If you can provide >> me with the power draw and a description of the hardware, that would be >> great. >> >> I have some numbers from a friend who lurks on this list, but the more >> data points I have, the better I can justify my power requirements for a >> new cluster purchase I'm working on. >> > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf ------------------------------ Subject: Digest Footer _______________________________________________ Beowulf mailing list Beowulf@beowulf.org http://www.beowulf.org/mailman/listinfo/beowulf ------------------------------ End of Beowulf Digest, Vol 125, Issue 13 **************************************** _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
