I did do some testing with the original round of updates and I found the supplementary GPFS monitoring tools (mmsysmon) written in Python were having a profoundly negative affect on performance with the Spectre/Meltdown patches with even first pass things like Linpack. You can disable the supplementary monitoring though and that cleared up the performance issues I was seeing. I didn't get very deep into it before they retracted all the firmware updates and we decided to wait and see. IBM had talked about providing options to make the querying less intrusive and less frequent on the GPFS UG mailing list in newer versions of GPFS as well.
Stephen ________________________________________ From: Beowulf <beowulf-boun...@beowulf.org> on behalf of Ryan Novosielski <novos...@rutgers.edu> Sent: Thursday, March 8, 2018 12:55:34 PM To: Beowulf List Subject: Re: [Beowulf] Update on dealing with Spectre and Meltdown I have been waiting, specifically, for Prentice Bisbal to try it. :-P I actually have not checked to see whether any other upgrades I’ve applied have handled these vulnerabilities (I’d assume RedHat would have patched their distribution). FWIW, however, I did receive a report that one of our newer images seemed slower. Haven’t gotten a chance to fully investigate those reports — could be any number of other things. -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novos...@rutgers.edu<mailto:novos...@rutgers.edu> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' On Mar 8, 2018, at 13:13, Alex Chekholko via Beowulf <beowulf@beowulf.org<mailto:beowulf@beowulf.org>> wrote: As an experienced and cynical HPC admin working with trusted users... I have not installed any kind of firmware or kernel updates. I'll wait for you to go first :) On Thu, Mar 8, 2018 at 10:10 AM, Prentice Bisbal <pbis...@pppl.gov<mailto:pbis...@pppl.gov>> wrote: Beowulfers, Have any of you updated the kernels on your clusters to fix the Spectre and Meltdown vulnerabilities? I was following this issue closely for the first couple of weeks. There seemed to be a lack of consensus on how much these fixed would impact HPC jobs, and if I recall correctly, some of the patches really hurt performance, or caused other problems. We took a wait-and-see approach here. So now that I've waited a while, what did you see? -- Prentice _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org<mailto:Beowulf@beowulf.org> sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.beowulf.org%2Fmailman%2Flistinfo%2Fbeowulf&data=02%7C01%7Cnovosirj%40rutgers.edu%7Cc4cd547515a346014d0d08d5852058a0%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C636561296368291958&sdata=4fTJudM1x6k1L3KPRsLITJEy%2FKI%2FqaXOGJ6qwDJmgG4%3D&reserved=0> _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org<mailto:Beowulf@beowulf.org> sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.beowulf.org%2Fmailman%2Flistinfo%2Fbeowulf&data=02%7C01%7Cnovosirj%40rutgers.edu%7Cc4cd547515a346014d0d08d5852058a0%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C636561296368291958&sdata=4fTJudM1x6k1L3KPRsLITJEy%2FKI%2FqaXOGJ6qwDJmgG4%3D&reserved=0 _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
