Re: [Beowulf] RoCE vs. InfiniBand

2020-11-26 Thread Chris Samuel
On Thursday, 26 November 2020 3:14:05 AM PST Jörg Saßmannshausen wrote: > Now, traditionally I would say that we are going for InfiniBand. However, > for reasons I don't want to go into right now, our existing file storage > (Lustre) will be in a different location. Thus, we decided to go for RoCE

Re: [Beowulf] Automatically replication of directories among nodes

2020-11-26 Thread James Braid
On Wed, 25 Nov 2020, 09:28 Lux, Jim (US 7140) via Beowulf, < beowulf@beowulf.org> wrote: > What I have is 3 rPi computers, A,B, and C, and what I’d like to do is > keep the desktop and some data directories on all of them synchronized. So > if on node A, I add something to A:~/Desktop, it (in sho

Re: [Beowulf] RoCE vs. InfiniBand

2020-11-26 Thread Lance Wilson via Beowulf
We are a ROCE shop with HPC on private cloud and have had a mostly good experience. We have had quite a number of issues over time that were bugs and needed vendor support to resolve. Some of which have taken a long time. So from a maturity perspective IB is definitely much better. As a matter of p

Re: [Beowulf] RoCE vs. InfiniBand

2020-11-26 Thread Gilad Shainer
Let me try to help: - OpenStack is supported natively on InfiniBand already, therefore there is no need to go to Ethernet for that - File system wise, you can have IB file system, and connect directly to IB system. - Depends on the distance, you can run 2Km IB betwee

Re: [Beowulf] RoCE vs. InfiniBand

2020-11-26 Thread John Hearns
Jorg, I think I might know where the Lustre storage is ! It is possible to install storage routers, so you could route between ethernet and infiniband. It is also worth saying that Mellanox have Metro Infiniband switches - though I do not think they go as far as the west of London! Seriously thoug

[Beowulf] RoCE vs. InfiniBand

2020-11-26 Thread Jörg Saßmannshausen
Dear all, as the DNS problems have been solve (many thanks for doing this!), I was wondering if people on the list have some experiences with this question: We are currently in the process to purchase a new cluster and we want to use OpenStack for the whole management of the cluster. Part of t