Will- Looks like even with the cluster limit I still get 3 discrete runs, when using the cluster example. I did a very simple play book and you can see the gathering_facts stages gets run in triplicate: [image: image.png] Definitely changed the behavior but not quite were I need it to go. However, it has given me an interesting ideas to try.
On Sun, Nov 26, 2023 at 3:06 PM Evan Hisey <[email protected]> wrote: > Will- > That was exactly the issue. I will give the bottom solution a go. I think > that will work, I will need to play with generating the group, but I think > t I can make it work. Thanks for the help, will update when I get something > working or fail :) > > > On Sun, Nov 26, 2023 at 11:25 AM Will McDonald <[email protected]> > wrote: > >> OK, after some experimentation, I think I see what your problem might be? >> If you do something like: >> >> BOX_IMAGE = "fedora/37-cloud-base" >> NODE_COUNT = 2 >> >> Vagrant.configure("2") do |config| >> >> (1..NODE_COUNT).each do |i| >> config.vm.define "node#{i}" do |subconfig| >> subconfig.vm.box = BOX_IMAGE >> subconfig.vm.hostname = "node#{i}" >> >> if i == NODE_COUNT >> config.vm.provision :ansible do |ansible| >> # Disable default limit to connect to all the machines >> ansible.limit = "all" >> ansible.playbook = "playbook.yml" >> end >> end >> >> end >> end >> >> end >> >> The Vagrant Ansible provisioner fires for every VM causing multiple >> discrete runs, you can control that to a degree with ansible.limit, the >> hosts statement in the playbook and/or delegate_to but it would be hard to >> do stateful cross-cluster config. >> >> If you do something like the following instead, this will provision all 3 >> Vagrant boxes and then fire the provisioner* once *triggering an Ansible >> run just for the final box: >> >> wmcdonald@fedora:~/working/vagrant/fedora-multi$ cat Vagrantfile >> Vagrant.configure(2) do |config| >> #Define the number of nodes to spin up >> N = 3 >> >> #Iterate over nodes >> (1..N).each do |node_id| >> nid = (node_id - 1) >> >> config.vm.define "node#{nid}" do |node| >> node.vm.box = "fedora/37-cloud-base" >> node.vm.provider "virtualbox" do |vb| >> vb.memory = "1024" >> end >> node.vm.hostname = "node#{nid}" >> >> if node_id == N >> node.vm.provision "ansible" do |ansible| >> ansible.limit = "all" >> ansible.groups = { >> "cluster-nodes" => [ >> "node0", >> "node1", >> "node2", >> ] >> } >> ansible.playbook = "playbook.yml" >> end >> end >> >> end >> end >> end >> >> wmcdonald@fedora:~/working/vagrant/fedora-multi130$ cat playbook.yml >> - name: Vagrant post-provision >> hosts: cluster_nodes >> >> tasks: >> - name: Debug vars for hosts >> debug: >> var: ansible_play_hosts >> >> Note that the provisioner will run once but still parallelise like a >> normal Ansible run would and hit each node because we're setting the hosts >> to the group members. You could further limit with delegate_to or have one >> cluster node in its own 'primary_node' group in addition to the >> cluster_nodes. >> >> See: >> https://everythingshouldbevirtual.com/automation/virtualization/vagrant-ansible-provisioning-multi-nodes/ >> And another variant with per-box behaviour here: >> https://stackoverflow.com/questions/54468546/how-to-run-an-ansible-playbook-on-a-specific-vagrant-host >> >> >> >> On Sun, 26 Nov 2023 at 00:22, Will McDonald <[email protected]> wrote: >> >>> There are a couple of ways you could exercise "control over the process >>> to pull data from host 1 to be used on host 2 and 3". >>> >>> If you look at >>> https://manski.net/2016/09/vagrant-multi-machine-tutorial/#multi-machine.3A-the-clever-way >>> 3 nodes are provisioned, one as primary, then two as secondary nodes and >>> it'd be relatively trivial to use this to key off the 'primary' node to do >>> what you needed, I imagine. >>> >>> Where I've had scenarios provisioning 3 nodes of something in a 2n+1 >>> cluster (basically anything like Mongo, Etcd, Zookeeper etc. etc.) and you >>> need to at least temporarily choose a semi-deterministic primary I've used >>> logic like: >>> >>> pre_tasks: >>> - name: pre_tasks | cluster member role setup for multiple hosts >>> block: >>> - name: pre_tasks | set cluster role to primary when >>> inventory_hostame matches random seed >>> set_fact: >>> cluster_role: primary >>> when: inventory_hostname == >>> ansible_play_hosts|random(seed=ansible_play_hosts | join()) >>> >>> - name: pre_tasks | set mongo replication role to secondary when >>> inventory_hostame does not match random seed >>> set_fact: >>> cluster_role: secondary >>> when: inventory_hostname != >>> ansible_play_hosts|random(seed=ansible_play_hosts | join()) >>> >>> - name: pre_tasks | create a custom facts.d directory on the target >>> host >>> file: >>> state: directory >>> recurse: true >>> path: /etc/ansible/facts.d >>> >>> - name: pre_tasks | persist the cluster membership role as a custom >>> fact >>> copy: >>> content: | >>> {'cluster_role':'{{ cluster_role }}'} >>> dest: /etc/ansible/facts.d/cluster.fact >>> mode: 0644 >>> owner: root >>> group: root >>> >>> *Warning! *This sets a *transient value* in facts.d. Which in my cases >>> is fine for our purposes. If your cluster membership state changes >>> post-setup, the fact would be misleading. (i.e. a node flaps and another >>> cluster member assumes leader/primary.) >>> >>> You would want to replace cluster.fact with something that dynamically >>> pulls out the cluster role membership state of a node once the >>> cluster/replicaset/whatever topology is provisioned and configured. >>> >>> >>> On Sat, 25 Nov 2023 at 23:25, Evan Hisey <[email protected]> wrote: >>> >>>> Definitely an edge case. Not an issue in my file atleast as written >>>> based on my understanding of the process, but possibly an issue in my >>>> understanding of how vagrant is executing ansible as it looks like vagrant >>>> runs on each vm as a separate job in either case, just in parallel on each >>>> the second time. I still need control over the process to pull data from >>>> host 1 to be used on host 2 and 3, which if it is running in parallel as >>>> multiple jobs would still be an issue. If it in fact runs a single ansible >>>> playbook across the inventory, then that could work, and be the opposite of >>>> how I am understanding vagrant ansible provider works. I would need to >>>> refactor a large chunk of the application code to support that, but that >>>> can be easily done. >>>> >>>> On Sat, Nov 25, 2023 at 4:44 PM Will McDonald <[email protected]> >>>> wrote: >>>> >>>>> I think you may be misunderstanding me, or I'm misunderstanding you. >>>>> >>>>> Just for clarity's sake, the flow you would like is: >>>>> >>>>> 1. An Ansible control node runs a playbook (or role) on >>>>> <controlnode> targeting a machine, <targetmachine> >>>>> 2. The <targetmachine> is configured to run as a Vagrant host with >>>>> a virtualisation provider (Virtualbox, Libvirt or whatever) in order to >>>>> support Vagrant box creation >>>>> 3. You then have a Vagrantfile which runs on <targetmachine> and >>>>> configures multiple Vagrant boxes <vb1>, <vb2>, <vb3> >>>>> 4. Once <vb1>, <vb2>, <vb3> are UP* and only then,* you want to >>>>> run some Ansible which needs the primary and 2 secondaries to be up >>>>> >>>>> That being the case, then that is the behaviour that >>>>> https://developer.hashicorp.com/vagrant/docs/provisioning/ansible#ansible-parallel-execution >>>>> describes. It's slightly poorly worded but to me: >>>>> >>>>> # Only execute once the Ansible provisioner, # when all the >>>>> machines are up and ready. >>>>> >>>>> Is equivalent to: >>>>> >>>>> # Provision all Vagrant boxes in the multi-machine setup. >>>>> # Only once all the machines are up and ready, run the Ansible >>>>> provisioner >>>>> >>>>> If that's not what's happening, that's likely a Vagrant configuration >>>>> or provisioner misbehaviour? >>>>> >>>>> That's why I'm saying this isn't necessarily an Ansible thing. That >>>>> wording, the boxes should all spin up before any Vagrant Ansible >>>>> provisioner runs, you're saying that's not the case. That sounds like >>>>> either your Vagrantfile is wrong, or your Vagrant VM provisioner or >>>>> something else isn't working as expected. >>>>> >>>>> I'm spinning this up on a test but if you already have a test >>>>> case/reproducer, or can provide more info on your Vagrant setup then this >>>>> would collectively help people help you. If there's an obvious error in >>>>> your Vagrantfile it could be a simple fix rather than an edge case. >>>>> >>>>> cf: >>>>> - >>>>> https://manski.net/2016/09/vagrant-multi-machine-tutorial/#multi-machine.3A-the-clever-way >>>>> - >>>>> https://developer.hashicorp.com/vagrant/docs/provisioning/ansible#ansible-parallel-execution >>>>> >>>>> >>>>> On Sat, 25 Nov 2023 at 21:55, Evan Hisey <[email protected]> wrote: >>>>> >>>>>> Vagrant is behaving fine, so not a vagrant specific problem. It is a >>>>>> task problem. I need the vagrant hosts fully installed first because I >>>>>> have >>>>>> to collect data from all 3 at once before deploying the software, and >>>>>> during software deployment I have to install the master first, collect >>>>>> keys >>>>>> and then install the slaves. Vagrant provider provisions does provide >>>>>> this >>>>>> kind of control as it assumes the each provisioned VM is self contained. >>>>>> A >>>>>> more typical solution would be to directly remote in to the VM's for >>>>>> ansible to run after deployment from the remote controller, but that is >>>>>> not >>>>>> an available option. Only the vagrant host will have access to the >>>>>> vagrant >>>>>> vms, and really only as the vagrant user. The last limitation is not hard >>>>>> to deal with, as vagrant provides everything an ansible job would need if >>>>>> run from the vagrant host. >>>>>> >>>>>> That is why I need to trigger to a vagrant host ansible playbook, >>>>>> since it can't not run from the initial ansible controller. Yes it is a >>>>>> bit >>>>>> of an odd edge case, as the vagrant provider normally would be plenty. >>>>>> >>>>>> On Sat, Nov 25, 2023 at 2:08 PM Will McDonald <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> It sounds like a Vagrant issue rather than an Ansible issue. Or >>>>>>> possibly a niche Vagrant provider problem. >>>>>>> >>>>>>> Can you share a sample Vagrantfile that's not behaving as it should >>>>>>> and details of the target OS of the Vagrant host, and the virtualisation >>>>>>> provider you're using? >>>>>>> >>>>>>> >>>>>>> On Sat, 25 Nov 2023 at 19:30, Evan Hisey <[email protected]> wrote: >>>>>>> >>>>>>>> Already tried it and it does not work, which was why I explicitly >>>>>>>> referenced that behavior as not working as not working in this >>>>>>>> scenario.While vagrant can run playbooks at provisioning time. it does >>>>>>>> not >>>>>>>> really proivde a way to to control when the provisioin runs. All 3 >>>>>>>> hosts >>>>>>>> need to be up be for the first host can be provisioned since it >>>>>>>> requires >>>>>>>> the ips of the later hosts. Second option does not work, as the remote >>>>>>>> control node does not have access to the VMs, as mentioned. Which is >>>>>>>> what >>>>>>>> lead to the need to trigger a second playbook. otherwise could lust >>>>>>>> load >>>>>>>> the vagrant generated inventory with add_host module. >>>>>>>> >>>>>>>> IC ould do some ugly sequencing of the "vagrant up --provision" >>>>>>>> from a playbook to control the ansible provisioning sequence of the >>>>>>>> vms, >>>>>>>> but I am trying to avoid using ugly shell commands as much as I can. >>>>>>>> If I >>>>>>>> uses a shell command I could also just trigger an ansible playbook >>>>>>>> that >>>>>>>> way, but feels wrong. >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Nov 25, 2023 at 12:40 PM Will McDonald <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Quickly skimming the Vagrant Ansible provisioner docs, isn't this >>>>>>>>> precisely the behaviour you're looking for: >>>>>>>>> >>>>>>>>> >>>>>>>>> https://developer.hashicorp.com/vagrant/docs/provisioning/ansible#ansible-parallel-execution >>>>>>>>> >>>>>>>>> # Only execute once the Ansible provisioner, # when all the >>>>>>>>> machines are up and ready. >>>>>>>>> >>>>>>>>> So you would spin up all your Vagrant boxes from your control >>>>>>>>> node, wait for that to complete, template out a static inventory of >>>>>>>>> your >>>>>>>>> Vagrant boxes then run your subsequent Vagrant Ansible provisioner >>>>>>>>> automation? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sat, 25 Nov 2023 at 18:20, Evan Hisey <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> I am working on a scenario where the first playbook executes >>>>>>>>>> commands on a remote host to create a vagrant host and spins up >>>>>>>>>> multiple >>>>>>>>>> vms. Vagrant can triggers it's own ansible provisioning runs but >>>>>>>>>> they are >>>>>>>>>> only single host aware and run when the host is provisioned. That >>>>>>>>>> does not >>>>>>>>>> work in this case, as I need all VM's running BEFORE the deployment >>>>>>>>>> playbook can be triggered. Added wrinkle is the VMs are accessible >>>>>>>>>> at this >>>>>>>>>> time from outside the vagrant host. If they were, I could simply >>>>>>>>>> import the >>>>>>>>>> vagrant host list into the controller inventory and refresh. >>>>>>>>>> >>>>>>>>>> Right now I am looking at the possibility of using >>>>>>>>>> ansible.builtin.shell to trigger the new ansible-playbook command on >>>>>>>>>> the >>>>>>>>>> vagrant host to run the vagrant VM application configuration. But >>>>>>>>>> while >>>>>>>>>> this works it is not exactly ansible clean. Suggestions on >>>>>>>>>> approaches? >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Evan Hisey >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "Ansible Project" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/ansible-project/CAEcFzYwsmYmvh%3DWJwSNmJWertkxFRDiKkumnwhzAFupggP58Vg%40mail.gmail.com >>>>>>>>>> <https://groups.google.com/d/msgid/ansible-project/CAEcFzYwsmYmvh%3DWJwSNmJWertkxFRDiKkumnwhzAFupggP58Vg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Ansible Project" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/ansible-project/CAKtKohS2LdL4qtTyQF8VdV6pu2tjL3mx868TgvkwvAEUkRbF%3Dg%40mail.gmail.com >>>>>>>>> <https://groups.google.com/d/msgid/ansible-project/CAKtKohS2LdL4qtTyQF8VdV6pu2tjL3mx868TgvkwvAEUkRbF%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Ansible Project" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/ansible-project/CAEcFzYxdk%3DfA13__d1wEJTiTkpDnaR2eqEpGm%2BXy3v3L1M%3DJ9A%40mail.gmail.com >>>>>>>> <https://groups.google.com/d/msgid/ansible-project/CAEcFzYxdk%3DfA13__d1wEJTiTkpDnaR2eqEpGm%2BXy3v3L1M%3DJ9A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Ansible Project" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/ansible-project/CAKtKohQ8PP4n7h-ERfE4iONYxZK-nZRLaoMwvRATzSj1hnD%3Dbw%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/ansible-project/CAKtKohQ8PP4n7h-ERfE4iONYxZK-nZRLaoMwvRATzSj1hnD%3Dbw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Ansible Project" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/ansible-project/CAEcFzYwxGLhF275kMbBaMMUx6FNQoduaxO7wa_j9AVkUw5ZQVg%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/ansible-project/CAEcFzYwxGLhF275kMbBaMMUx6FNQoduaxO7wa_j9AVkUw5ZQVg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Ansible Project" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/ansible-project/CAKtKohS9rcVZ0rOe_MH9bdb2EgFL7jeBmqwLgyRbZMG2NjmrmQ%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/ansible-project/CAKtKohS9rcVZ0rOe_MH9bdb2EgFL7jeBmqwLgyRbZMG2NjmrmQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Ansible Project" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/ansible-project/CAEcFzYzr7mYhJJimRKsyoV_ikYQoxjHYgOs-pABj4WEPUfoMbg%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/ansible-project/CAEcFzYzr7mYhJJimRKsyoV_ikYQoxjHYgOs-pABj4WEPUfoMbg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Ansible Project" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/ansible-project/CAKtKohSGMsrKNf9rAqafEk3UoLf9AdWK9G491JB87gaAUhUEmg%40mail.gmail.com >> <https://groups.google.com/d/msgid/ansible-project/CAKtKohSGMsrKNf9rAqafEk3UoLf9AdWK9G491JB87gaAUhUEmg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Ansible Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAEcFzYyhyZ%2BzxG2QX0NGySx4iDgydg41Mbeubu5JEpKpLOOccg%40mail.gmail.com.
