I am going to answer multiple points below:
On 1/3/20 11:36 pm, Daniel Frey wrote:
On 3/1/20 6:33 AM, Rich Freeman wrote:
On Sun, Mar 1, 2020 at 2:13 AM William Kenworthy <[email protected]>
wrote:
Keep in mind that rpi are not the only cheap, capable arm hardware out
there.
I am in Oz, delivery from HardKernel (the South Korean company behind
the Odroid line) takes ~1 week. Shipping is mostly via FedEx, who are a
bit pricy and for me means a 30m drive to get it from a delivery centre
as I cant be home to take delivery - though the last delivery used a
different, much more flexible delivery company.
lizardfs and moosefs are very similar (originally a fork) - I went with
moosefs as the community was better a few months ago - but lizardfs
sounds like its getting back on track after some infighting (moosefs
apparently went through the same thing) I find the moosefs
documentation and community help are adequate, but ultimately they are
are a commercial project with a free taster offering hence some
limitations in design for the community version (such as single
master/no shadow masters) that lizardfs doesn't have. I will move to
lizardfs when I am satisfied they have their act together because of
this - having a single master means taking the whole cluster offline
when the master needs maintenance or fails which is painful
For those wanting to run a lot of drives on a single host - that defeats
the main advantage of using a chunkserver based filesystem -
redundancy. Its far more common to have a host fail than a disk drive.
Losing the major part of your storage in one go means the cluster is
effectively dead - hence having a lot of completely separate systems is
much more reliable - yes, I did try having 4 sata drives on an atom
board and found it was easy to justify two more HC2's for the
reliability. (note, its not the effect of the atom boards reliability I
am pointing out, but the effect on the whole storage system of losing
such a large percentage of capacity in one go - think maintenance, fail
to startup etc. - its more common than you may think)
Dale: there is no single reference - I just designed on the fly and did
what was necessary to move from my existing KVM/Qemu architecture on a
two powerfull intel systems, each with 8TB storage to an lxc container
based system backed by moosefs. It uses an odroid arm based n2 for lxc,
an arm based xu4 for management and software compilation (its not that
powerfull, but is adequate), the HC2's (very similar to the xu4 - same
arm architecture) and an H2 to run the mfsmaster and I am moving to
ansible for management. Overall, I am finding less maintenance due to
better backups and better reliability, considerably lower power
consumption while overall performance seems better.
Gotchas:
1. I originally got the 4gb ram N2 to run the mfsmaster software - tests
were excellent - until I added multiple copies of a mailserver with
nearly 20 years of history - hit swap and slowed to a crawl. So I now
have an intel based Odroid H2 with 32GB ram - currently is using ~4.2GB
ram for 7TB of 20TB used, but has hit over 32GbB and well into swap
while converging. One admin in our local LUG is using 4xHC2 with the
master running on one of the HC2's as a media server with no problems -
its millions of small files added ina short time that causes the grief -
there is a formula in the moosefs documentation allowing resources to be
estimated - on the mailing list I saw a mention of a data centre using
something like 150G ram and having problems!
2. The Odroid HC2 has a single sata port and a single usb port - I have
5 of them, two have a sata + a usb3 drive attached. I did try (on the
N2) using multiple usb3 drives on the internal hub - disaster with way
too much traffic through the hub - don't do it!
3. Storage options are an SD card (the faster the better - swap on an SD
card ... sucks!) or eMMC (5xfaster than even a good SD card.) for the N2
and xu4 - HC2's can only do sdcard, or sata. The H2 can do sd card,
sata, eMMC, or m2 NVME - this last really flies! Note that almost all
arm system max out at 2 to 4GB of ram, so swap is usually needed for
safety - depending on OOM killer resource management on a SAN type
storage system is asking for corruption like I saw in one recent gentoo
email.
4. xu4/HC2 are 32 bit arm v7, the N2 is 64 bit and runs aarch64 nicely -
I copied my rpi images across repurposed them (hooray for
emerge/portage!). I do not use, or have done any work on their
graphics/multimedia capabilities
5. I want to move to all gentoo-sources kernels - the xu4/HC2's are
still on the odroid kernel until I get around to it. The N2 was a lot
of work, but ultimately successful, the H2 is standard amd64 EFI.
Have fun!
BillK
