I'm surprised no one here has mentioned tuning kernel/network parameters. I would take at which of these parameters you can tune to improve performance first because it's free, quick, and the least labor-intensive way to improve performance. I would take a look at the website below and see what parameters you can tweak to improve your performance.

https://fasterdata.es.net/

Prentice

On 8/10/23 3:35 PM, Jeff Johnson wrote:
Leo,

NFS can be a hindrance but if tuned and configured properly might not be as terrible. Some thoughts...

  * What interface are the nodes accessing NFS via? Ethernet or
    Infiniband?
  * Have you tuned the number of NFS server threads above defaults?
  * As a test, you could deploy a single Lustre node that would act as
    MGS/MDS and OSS simultaneously to test for performance gains via
    Infiniband.
  * Your scratch volume must really be scratch because you are
    running with no parity protection (two disk os SSD stripe)
  * You're probably better off with tuned NFS as opposed to GlusterFS

--Jeff

On Thu, Aug 10, 2023 at 12:19 PM leo camilo <lhcam...@gmail.com> wrote:

    Hi everyone,

    I was hoping I would seek some sage advice from you guys.

    At my department we have build this small prototyping cluster with
    5 compute nodes,1 name node and 1 file server.

    Up until now, the name node contained the scratch partition, which
    consisted of 2x4TB HDD, which form an 8 TB striped zfs pool. The
    pool is shared to all the nodes using nfs. The compute nodes and
    the name node and compute nodes are connected with both cat6
    ethernet net cable and infiniband. Each compute node has 40 cores.

    Recently I have attempted to launch computation from each node (40
    tasks per node), so 1 computation per node.  And the performance
    was abysmal. I reckon I might have reached the limits of NFS.

    I then realised that this was due to very poor performance from
    NFS. I am not using stateless nodes, so each node has about 200 GB
    of SSD storage and running directly from there was a lot faster.

    So, to solve the issue,  I reckon I should replace NFS with
    something better. I have ordered 2x4TB NVMEs  for the new scratch
    and I was thinking of :

      * using the 2x4TB NVME in a striped ZFS pool and use a single
        node GlusterFS to replace NFS
      * using the 2x4TB NVME with GlusterFS in a distributed
        arrangement (still single node)

    Some people told me to use lustre,but I reckon that might be
    overkill. And I would only use a single fileserver machine(1 node).

    Could you guys give me some sage advice here?

    Thanks in advance


    _______________________________________________
    Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin
    Computing
    To change your subscription (digest mode or unsubscribe) visit
    https://beowulf.org/cgi-bin/mailman/listinfo/beowulf



--
------------------------------
Jeff Johnson
Co-Founder
Aeon Computing

jeff.john...@aeoncomputing.com
www.aeoncomputing.com <http://www.aeoncomputing.com>
t: 858-412-3810 x1001   f: 858-412-3845
m: 619-204-9061

4170 Morena Boulevard, Suite C - San Diego, CA 92117

High-Performance Computing / Lustre Filesystems / Scale-out Storage

_______________________________________________
Beowulf mailing list,Beowulf@beowulf.org  sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) 
visithttps://beowulf.org/cgi-bin/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf

Reply via email to