Re: [Beowulf] Suggestions to what DFS to use

Hi John, On 15/02/17 17:33, John Hanks wrote: > So "clusters" is a strong word, we have a collection of ~22,000 cores of > assorted systems, basically if someone leaves a laptop laying around > unprotected we might try to run a job on it. And being bioinformatic-y, > our problem with this and all

Re: [Beowulf] Suggestions to what DFS to use

So "clusters" is a strong word, we have a collection of ~22,000 cores of assorted systems, basically if someone leaves a laptop laying around unprotected we might try to run a job on it. And being bioinformatic-y, our problem with this and all storage is metadata related. The original procurement d

Re: [Beowulf] Suggestions to what DFS to use

On 15/02/17 17:03, John Hanks wrote: > When we were looking at a possible GPFS client license purchase we ran > the client on our nodes and did some basic testing. The client did give > us a bit of a boost in performance over NFS, but still we could tip GPFS > over with a small fraction of our ava

Re: [Beowulf] Suggestions to what DFS to use

When we were looking at a possible GPFS client license purchase we ran the client on our nodes and did some basic testing. The client did give us a bit of a boost in performance over NFS, but still we could tip GPFS over with a small fraction of our available nodes. The improvement was not enough t

Re: [Beowulf] solaris?

It has a minor role for us for storage (ZFS), but we're retiring our Solaris boxes as quickly as we can in favor of more GPFS. Skylar On 02/14/2017 01:28 PM, Michael Di Domenico wrote: > just out of morbid curiosity, does Solaris even have a stake in HPC > anymore? I've not heard boo about it in

Re: [Beowulf] Suggestions to what DFS to use

On 13/02/17 18:55, Tony Brian Albers wrote: > I've looked at MooseFS which seems to be able to do the trick, but are > there any others that might do? There are some folks elsewhere at the university here that are looking at CephFS, so I'd be glad to hear about any experiences with that. They'r

Re: [Beowulf] Suggestions to what DFS to use

On 02/14/2017 08:02 PM, Bogdan Costescu wrote: I can second the recommendation for BeeGFS. We have it in use for ~4 years with very good results, by now on 3 different FSes. We also run I'll freely admit to being biased here, but BeeGFS is definitely something you should be evaluating/using.

Re: [Beowulf] Suggestions to what DFS to use

On 14/02/17 18:31, John Hanks wrote: > > 1. (~500 TB) DDN SFA12K running gridscaler (GPFS) but without GPFS > clients on nodes, this is presented to the cluster through cNFS. [...] > Depending on your benchmark, 1, 2 or 3 may be faster. GPFS falls over > wheezing under load. I suspect that's more

Re: [Beowulf] Suggestions to what DFS to use

I can second the recommendation for BeeGFS. We have it in use for ~4 years with very good results, by now on 3 different FSes. We also run it on SuperMicro hardware and Infiniband, but use the "classic" combination with ext4 for metadata and xfs for storage servers - of course, with RAID controller

Re: [Beowulf] Suggestions to what DFS to use

If I can help, I'm inside IBM. I'm the marketing lead for IBM Spectrum Scale (aka GPFS), but I have solid connections to the field tech support and development teams. my corporate email is dougla...@us.ibm.com IBM just announced that HortonWorks will be supported on IBM Spectrum Scale. IBM has a

Re: [Beowulf] solaris?

Lots going on related to Oracle and Solaris recently: https://www.thelayoff.com/t/KTCW4qz If anyone was considering Solaris before, they probably aren’t now. Sounds like they killed Solaris 12, which can’t be good for Solaris. > On Feb 14, 2017, at 4:28 PM, Michael Di Domenico > wrote: > > j

Re: [Beowulf] solaris?

On 02/14/2017 04:28 PM, Michael Di Domenico wrote: just out of morbid curiosity, does Solaris even have a stake in HPC anymore? I've not heard boo about it in quite awhile and there doesn't appear to be even one system on the top500 running it. Solaris is effectively dead (IMO). SmartOS is

[Beowulf] solaris?

just out of morbid curiosity, does Solaris even have a stake in HPC anymore? I've not heard boo about it in quite awhile and there doesn't appear to be even one system on the top500 running it. ___ Beowulf mailing list, Beowulf@beowulf.org sponsored by P

Re: [Beowulf] Suggestions to what DFS to use

Of course there are tons of options depending upon what you want and your IO patterns of the applications. Doug's comments about HDFS are great - he's a very good expert in this area. Depending upon your IO patterns and workload, NFS may work well. I've found it work quite well unless you have a

Re: [Beowulf] Suggestions to what DFS to use

Should have included this in my last message: https://github.com/zfsonlinux/zfs/wiki/RHEL-%26-CentOS One other aspect of ZFS I overlooked in my earlier messages is the built in compression. At one point I backed up 460TB of data from our GPFS system onto ~300TB of space on a ZFS system using gzip

Re: [Beowulf] Suggestions to what DFS to use

All our nodes, even most of our fileservers (non-DDN), boot statelessly (warewulf) and all local disks are managed by ZFS, either with JBOD controllers or with non-JBOD controllers configuring each disk as a 1 drive RAID0. So if at all possible, ZFS gets control of the raw disk. ZFS has been extre

Re: [Beowulf] Suggestions to what DFS to use

On 2017-02-14 11:44, Jörg Saßmannshausen wrote: > Hi John, > > thanks for the very interesting and informative post. > I am looking into large storage space right now as well so this came really > timely for me! :-) > > One question: I have noticed you were using ZFS on Linux (CentOS 6.8). What > a

Re: [Beowulf] Suggestions to what DFS to use

On 2017-02-14 03:00, Douglas Eadline wrote: > >> Hi guys, >> >> So, we're running a small(as in a small number of nodes(10), not >> storage(170TB)) hadoop cluster here. Right now we're on IBM Spectrum >> Scale(GPFS) which works fine and has POSIX support. On top of GPFS we >> have a GPFS transparen

Re: [Beowulf] Suggestions to what DFS to use

On 2017-02-14 02:36, Skylar Thompson wrote: > Is there anything in particular that is causing you to move away from GPFS? > > Skylar > > On 02/12/2017 11:55 PM, Tony Brian Albers wrote: >> Hi guys, >> >> So, we're running a small(as in a small number of nodes(10), not >> storage(170TB)) hadoop clus

Re: [Beowulf] Suggestions to what DFS to use

On 2017-02-13 20:45, Ellis H. Wilson III wrote: > On 02/13/17 14:00, Greg Lindahl wrote: >> On Mon, Feb 13, 2017 at 07:55:43AM +, Tony Brian Albers wrote: >>> Hi guys, >>> >>> So, we're running a small(as in a small number of nodes(10), not >>> storage(170TB)) hadoop cluster here. Right now we'

Re: [Beowulf] Suggestions to what DFS to use

On 2017-02-13 17:54, John Hanks wrote: > We've had pretty good luck with BeeGFS lately running on SuperMicro > vanilla hardware with ZFS as the underlying filesystem. It works pretty > well for the cheap end of the hardware spectrum and BeeGFS is free and > pretty amazing. It has held up to abuse u

Re: [Beowulf] Suggestions to what DFS to use

On 2017-02-13 17:30, Justin Y. Shi wrote: > Maybe you would consider Scality (http://www.scality.com/) for your > growth concerns. If you need speed, DDN is faster in rapid data > ingestion and for extreme HPC data needs. > > Justin > > On Mon, Feb 13, 2017 at 4:32 AM, Tony Brian Albers

Re: [Beowulf] Suggestions to what DFS to use

Hi John, thanks for the very interesting and informative post. I am looking into large storage space right now as well so this came really timely for me! :-) One question: I have noticed you were using ZFS on Linux (CentOS 6.8). What are you experiences with this? Does it work reliable? How di