On 29/6/20 5:09 pm, Jörg Saßmannshausen wrote:
we are currently planning a new cluster and this time around the idea was to
use OpenStack for the HPC part of the cluster as well.
I was wondering if somebody has some first hand experiences on the list here.
At $JOB-2 I helped a group set up a cluster on OpenStack (they were
resource constrained, they had access to OpenStack nodes and that was
it). In my experience it was just another added layer of complexity for
no added benefit and resulted in a number of outages due to failures in
the OpenStack layers underneath.
Given that Slurm which was being used there already had mature cgroups
support there really was no advantage to them to having a layer of
virtualisation on top of the hardware, especially as (if I'm remembering
properly) in the early days the virtualisation layer didn't properly
understand the Intel CPUs we had and so didn't reflect the correct
capabilities to the VM.
All that said, these days it's likely improved, and I know then people
were thinking about OpenStack "Ironic" which was a way for it to manage
bare metal nodes.
But I do know the folks in question eventually managed to go to purely a
bare metal solution and seemed a lot happier for it.
As for IB, I suspect that depends on the capabilities of your
virtualisation layer, but I do believe that is quite possible. This
cluster didn't have IB (when they started getting bare metal nodes they
went RoCE instead).
All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
