As the definitely-not-proud owner of a 2016 purchase of a 60-bay disk shelf attached to a single server with an Infiniband connection back to 54 compute nodes, NFS on spinning disks can definitely handle 5 40-core jobs, but your particular setup really can’t. Mine has hit its limits at times as well, but it’s about the IOPS from the disk array, the speed of the SAS cable connecting the disk shelf to the server, everything *but* NFS itself.
Swapping to NVMe should make a world of difference on its own, as long as you don’t have a bottleneck of 1 Gb Ethernet between your storage and the compute capacity. From: Beowulf <beowulf-boun...@beowulf.org> on behalf of leo camilo <lhcam...@gmail.com> Date: Thursday, August 10, 2023 at 3:04 PM To: Jeff Johnson <jeff.john...@aeoncomputing.com> Cc: Bernd Schubert <bernd.schub...@fastmail.fm>, Beowulf@beowulf.org <Beowulf@beowulf.org> Subject: Re: [Beowulf] NFS alternative for 200 core compute (beowulf) cluster External Email Warning This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests. ________________________________ Awesome, thanks for the info! Best, leo On Thu, 10 Aug 2023 at 22:01, Jeff Johnson <jeff.john...@aeoncomputing.com<mailto:jeff.john...@aeoncomputing.com>> wrote: Leo, Both BeeGFS and Lustre require a backend file system on the disks themselves. Both Lustre and BeeGFS support ZFS backend. --Jeff On Thu, Aug 10, 2023 at 1:00 PM leo camilo <lhcam...@gmail.com<mailto:lhcam...@gmail.com>> wrote: Hi there, thanks for your response. BeeGFS indeed looks like a good call option, though realistically I can only afford to use a single node/server for it. Would it be feasible to use zfs as volume manager coupled with BeeGFS for the shares, or should I write zfs off all together? thanks again, best, leo On Thu, 10 Aug 2023 at 21:29, Bernd Schubert <bernd.schub...@fastmail.fm<mailto:bernd.schub...@fastmail.fm>> wrote: On 8/10/23 21:18, leo camilo wrote: > Hi everyone, > > I was hoping I would seek some sage advice from you guys. > > At my department we have build this small prototyping cluster with 5 > compute nodes,1 name node and 1 file server. > > Up until now, the name node contained the scratch partition, which > consisted of 2x4TB HDD, which form an 8 TB striped zfs pool. The pool is > shared to all the nodes using nfs. The compute nodes and the name node > and compute nodes are connected with both cat6 ethernet net cable and > infiniband. Each compute node has 40 cores. > > Recently I have attempted to launch computation from each node (40 tasks > per node), so 1 computation per node. And the performance was abysmal. > I reckon I might have reached the limits of NFS. > > I then realised that this was due to very poor performance from NFS. I > am not using stateless nodes, so each node has about 200 GB of SSD > storage and running directly from there was a lot faster. > > So, to solve the issue, I reckon I should replace NFS with something > better. I have ordered 2x4TB NVMEs for the new scratch and I was > thinking of : > > * using the 2x4TB NVME in a striped ZFS pool and use a single node > GlusterFS to replace NFS > * using the 2x4TB NVME with GlusterFS in a distributed arrangement > (still single node) > > Some people told me to use lustre,but I reckon that might be overkill. > And I would only use a single fileserver machine(1 node). > > Could you guys give me some sage advice here? > So glusterfs is using fuse, which doesn't have the best performance reputation (although hopefully not for long - feel free to search for "fuse" + "uring"). If you want to avoid complexity of Lustre, maybe look into BeeGFS. Well, I would recommend to look into it anyway (as former developer I'm biased again ;) ). Cheers, Bernd _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org<mailto:Beowulf@beowulf.org> sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf -- ------------------------------ Jeff Johnson Co-Founder Aeon Computing jeff.john...@aeoncomputing.com<mailto:jeff.john...@aeoncomputing.com> www.aeoncomputing.com<http://www.aeoncomputing.com/> t: 858-412-3810 x1001 f: 858-412-3845 m: 619-204-9061 4170 Morena Boulevard, Suite C - San Diego, CA 92117 High-Performance Computing / Lustre Filesystems / Scale-out Storage
_______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
