Good morning ...
On 09/05/2017 01:24 AM, Stu Midgley wrote:
Morning everyone
I am in the process of redeveloping our cluster deployment and config
management environment and wondered what others are doing?
First, everything we currently have is basically home-grown.
Nothing wrong with this, if it adequately solves the problem. Many of
the frameworks people use for these things are highly opinionated, and
often, you'll find their opinions grate on your expectations. At
$dayjob-1, I developed our own kit precisely because so many of the
other toolkits did little to big things wrong; not simply from an
opinion point of view, but actively made specific errors that the
developers glossed over as that aspect was unimportant to them ... while
being of critical importance to me and my customers at the time.
Our cluster deployment is a system that I've developed over the years
and is pretty simple - if you know BASH and how pxe booting works. It
has everything from setting the correct parameters in the bios, zfs
ram disks for the OS, lustre for state files (usually in /var) - all
in the initrd.
We use it to boot cluster nodes, lustre servers, misc servers and
desktops.
We basically treat everything like a cluster.
The most competent baked distro out there for this was (in the past,
haven't used it recently) Warewulf. See https://github.com/warewulf/ .
Still under active development, and Greg and team do a generally great
job. Least opinionated distro around, most flexible, and some of the
best tooling.
However... we do have a proliferation of images... and all need to be
kept up-to-date and managed. Most of the changes from one image to
the next are config files.
Ahhh ... One of the things we did with our toolchain (it is open source,
I've just never pushed it to github) was to completely separate booting
from configuration. That is, units booted to an operational state
before we applied configuration. This was in part due to long
experience with nodes hanging during bootup with incorrect
configurations. If you minimize the chance for this, your nodes
(barring physical device failure) always boot. The only specific
opinion we had w.r.t. this system was that the nodes had to be bootable
via PXE, and there fore a working dhcp needed to exist on the network.
Post boot configuration, we drove via a script that downloaded and
launched other scripts. Since we PXE booted, network addresses were
fine. We didn't even enforce final network address determination on PXE
startup.
We looked at the booting process as a state machine. Lower level was
raw hardware, no power. Subsequent levels were bios POST, PXE of
kernel, configuration phase. During configuration phase *everything*
was on the table w.r.t. changes. We could (and did) alter networking,
using programmatic methods, databases, etc. to determine and configure
final network configs. Same for disks, and other resources.
Configuration changes could be pushed post boot by updating a script and
either pushing (not normally recommended for clusters of reasonable
size) or triggering a pull/run cycle for that script/dependencies.
This allowed us to update images and configuration asynchronously.
We had to manage images, but this turned out to be generally simple. I
was in the midst of putting image mappings into a distributed object
store when the company died. Config store is similarly simple, again
using the same mechanisms, and could be driven entirely programmatically.
Of course, for the chef/puppet/ansible/salt/cloudformation/... people,
we could drive their process as well.
We don't have a good config management (which might, hopefully, reduce
the number of images we need). We tried puppet, but it seems everyone
hates it. Its too complicated? Not the right tool?
Highly opinionated config management is IMO (and yes, I am aware this is
redundant humor) generally a bad idea. Config management that gets out
of your way until you need it is the right approach. Which is why we
never tried to dictate what config management our users would use. We
simply handled getting the system up to an operational state, and they
could use ours, theirs, or Frankensteinian kludges.
I was thinking of using git for config files, dumping a list of rpm's,
dumping the active services from systemd and somehow munging all that
together in the initrd. ie. git checkout the server to get config
files and systemctl enable/start the appropriate services etc.
It started to get complicated.
Any feedback/experiences appreciated. What works well? What doesn't?
IMO things that tie together config and booting are problematic at
scale. Leads to nearly unmanageable piles of images, as you've
experienced. Booting to an operational state, and applying all config
post boot (ask me about my fstab replacement some day), makes for a very
nice operational solution that scales wonderfully .... you can replicate
images to local image servers if you wish, replicate config servers,
load balance the whole thing to whatever scale you need.
Thanks.
--
Dr Stuart Midgley
sdm...@gmail.com <mailto:sdm...@gmail.com>
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
--
Joe Landman
e: joe.land...@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
l: https://www.linkedin.com/in/joelandman
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
