#AutomateAllTheThings

[datamattsson]

Managing VMware vSphere Virtual Machines with Ansible

I was tasked with extraordinary daunting task of provisioning a test environment on vSphere. I knew that the install was going to fail on me multiple times and I was in dire need of a few things:

Start over from inception - basically a blank sheet of paper

Create checkpoints and be able to revert to those checkpoints fairly easily

Do a ton of customization in the guest OS

The Anti-Pattern

I’ve been enslaved with vSphere in previous jobs. It’s a handy platform for various things. I was probably the first customer to run ESX on NetApp NFS fifteen years ago. I can vividly remember that already back then I was incredibly tired of “right clicking” in vCenter and I wrote extensive automation with the Perl bindings and SDKs that were available at the time. I get a rash if I have to do something manually in vCenter and I see it as nothing but an API endpoint. Manual work in vCenter is the worst TOIL and the anti-pattern of modern infrastructure management.

Hello Ansible

I manage my own environment, which is KVM based, entirely with Ansible. Sure, it’s statically assigned virtual machines but surprisingly, it works just great as I’m just deploying clusters where HA is dealt with elsewhere. When this project that I’m working on came up, I frantically started to map out everything I needed to do in the Ansible docs. Not too surprisingly, Ansible makes everything a breeze. You’ll find the VMware vSphere integration in the “Cloud Modules” section.

Inception

I needed to start with something. That includes some right-clicking in vCenter. I uploaded this vmdk file into one the datastores and manually configured a Virtual Machine template with the uploaded vmdk file. This is I could bear with as I only had to do it once. Surprisingly, I could not find a CentOS 7 OVA/OVF file that could deploy from (CentOS was requirement for the project, I’m an Ubuntu-first type of guy and they have plenty of images readily available).

Once you have that Virtual Machine template baked. Step away from vCenter, logout, close tab. Don���t look back (remember the name of the template!)

I’ve stashed the directory tree on GitHub. The Ansible pattern I prefer is that you use a ansible.cfg local to what you’re doing, playbooks to carry out your tasks and apply roles as necessary. I’m not going through the minutia of getting Ansible installed and all that jazz. The VMware modules have numerous Python dependences and they will tell you what is missing, simply pip install <whatever is complaining> to get rolling.

Going forward, let's assume:

git clone https://github.com/NimbleStorage/automation-examples cd cloud/vmware-vcenter

There are some variables that needs to be customized and tailored to any specific environment. The file that needs editing is host_vars/localhost that needs to be copied from host_vars/localhost-dist. Mine looks similar to this:

--- vcenter_hostname: 192.168.1.1 vcenter_username: administrator@vsphere.local vcenter_password: "HPE Cyber Security Will See This" vcenter_datastore: MY-DSX vcenter_folder: / vcenter_template: CentOS7 vcenter_datacenter: MY-DC vcenter_resource_pool: MY-RP # Misc config machine_group: machines machine_initial_user: root machine_initial_password: osboxes.org # Machine config machine_memory_mb: 2048 machine_num_cpus: 2 machine_num_cpu_cores_per_socket: 1 machine_networks: - name: VM Network - name: Island Network machine_disks: - size_gb: 500 type: thinProvisioned datastore: "{{ vcenter_datastore }}"

I also have a fairly basic inventory that I’m working with (in hosts):

[machines] tme-foo-m1 tme-foo-m2 tme-foo-m3 tme-foo-m4 tme-foo-m5 tme-foo-m6

Tailor your config and let’s move on.

Note: The network I’m sitting on is providing DHCP services with permanent leases and automatic DNS registration, I don’t have to deal with IP addressing. If static IP addressing is required, feel free to modify to your liking but I wouldn’t even know where to begin as the vmdk image I’m using as a starter is non-customizable.

Deploy Virtual Machines

First things first, provision the virtual machines. I intentionally didn’t want to screw around with VM snapshots to clone from. Full copies of each VM is being performed. I’m running this on the most efficient VMware storage array in the biz so I don’t really have to care that much about space.

Let’s deploy!

$ ansible-playbook deploy.yaml PLAY [localhost] **************************************************** TASK [Gathering Facts] ********************************************************************* Monday 04 November 2019 04:12:51 +0000 (0:00:00.096) 0:00:00.096 ******* ok: [localhost] TASK [deploy : Create a virtual machine from a template] ************************************************************** Monday 04 November 2019 04:12:52 +0000 (0:00:00.916) 0:00:01.012 ******* changed: [localhost -> localhost] => (item=tme-foo-m1) changed: [localhost -> localhost] => (item=tme-foo-m2) changed: [localhost -> localhost] => (item=tme-foo-m3) changed: [localhost -> localhost] => (item=tme-foo-m4) changed: [localhost -> localhost] => (item=tme-foo-m5) changed: [localhost -> localhost] => (item=tme-foo-m6) PLAY RECAP ********************************************************** localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 Monday 04 November 2019 04:31:37 +0000 (0:18:45.897) 0:18:46.910 ******* ===================================================================== deploy : Create a virtual machine from a template ---------- 1125.90s Gathering Facts ----------------------------------------------- 0.92s Playbook run took 0 days, 0 hours, 18 minutes, 46 seconds

In this phase we have machines deployed. They’re not very useful yet as I want to add my current SSH key from the machine I’m managing the environment from. Copy roles/prepare/files/authorized_keys-dist to roles/prepare/files/authorized_keys:

cp roles/prepare/files/authorized_keys-dist roles/prepare/files/authorized_keys

Add your public key to roles/prepare/files/authorized_keys. Also configure machine_user to match the username your managing your machines from.

Now, let’s prep the machines:

$ ansible-playbook prepare.yaml PLAY [localhost] **************************************************** TASK [Gathering Facts] ********************************************************************* Monday 04 November 2019 04:50:36 +0000 (0:00:00.102) 0:00:00.102 ******* ok: [localhost] TASK [prepare : Gather info about VM] ********************************************************************* Monday 04 November 2019 04:50:37 +0000 (0:00:00.889) 0:00:00.991 ******* ok: [localhost -> localhost] => (item=tme-foo-m1) ok: [localhost -> localhost] => (item=tme-foo-m2) ok: [localhost -> localhost] => (item=tme-foo-m3) ok: [localhost -> localhost] => (item=tme-foo-m4) ok: [localhost -> localhost] => (item=tme-foo-m5) ok: [localhost -> localhost] => (item=tme-foo-m6) TASK [prepare : Register IP in inventory] ********************************************************************* Monday 04 November 2019 04:50:41 +0000 (0:00:04.191) 0:00:05.183 ******* <very large blurb redacted> TASK [prepare : Test VM] ********************************************************************* Monday 04 November 2019 04:50:41 +0000 (0:00:00.157) 0:00:05.341 ******* ok: [localhost -> None] => (item=tme-foo-m1) ok: [localhost -> None] => (item=tme-foo-m2) ok: [localhost -> None] => (item=tme-foo-m3) ok: [localhost -> None] => (item=tme-foo-m4) ok: [localhost -> None] => (item=tme-foo-m5) ok: [localhost -> None] => (item=tme-foo-m6) TASK [prepare : Create ansible user] ********************************************************************* Monday 04 November 2019 04:50:46 +0000 (0:00:04.572) 0:00:09.914 ******* changed: [localhost -> None] => (item=tme-foo-m1) changed: [localhost -> None] => (item=tme-foo-m2) changed: [localhost -> None] => (item=tme-foo-m3) changed: [localhost -> None] => (item=tme-foo-m4) changed: [localhost -> None] => (item=tme-foo-m5) changed: [localhost -> None] => (item=tme-foo-m6) TASK [prepare : Upload new sudoers] ********************************************************************* Monday 04 November 2019 04:50:49 +0000 (0:00:03.283) 0:00:13.198 ******* changed: [localhost -> None] => (item=tme-foo-m1) changed: [localhost -> None] => (item=tme-foo-m2) changed: [localhost -> None] => (item=tme-foo-m3) changed: [localhost -> None] => (item=tme-foo-m4) changed: [localhost -> None] => (item=tme-foo-m5) changed: [localhost -> None] => (item=tme-foo-m6) TASK [prepare : Upload authorized_keys] ********************************************************************* Monday 04 November 2019 04:50:53 +0000 (0:00:04.124) 0:00:17.323 ******* changed: [localhost -> None] => (item=tme-foo-m1) changed: [localhost -> None] => (item=tme-foo-m2) changed: [localhost -> None] => (item=tme-foo-m3) changed: [localhost -> None] => (item=tme-foo-m4) changed: [localhost -> None] => (item=tme-foo-m5) changed: [localhost -> None] => (item=tme-foo-m6) PLAY RECAP ********************************************************** localhost : ok=9 changed=6 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 Monday 04 November 2019 04:51:01 +0000 (0:00:01.980) 0:00:24.903 ******* ===================================================================== prepare : Test VM --------------------------------------------- 4.57s prepare : Gather info about VM -------------------------------- 4.19s prepare : Upload new sudoers ---------------------------------- 4.12s prepare : Upload authorized_keys ------------------------------ 3.59s prepare : Create ansible user --------------------------------- 3.28s Gathering Facts ----------------------------------------------- 0.89s prepare : Register IP in inventory ---------------------------- 0.16s Playbook run took 0 days, 0 hours, 0 minutes, 20 seconds

At this stage, things should be in a pristine state. Let’s move on.

Managing Virtual Machines

The bleak inventory file what we have created should now be usable. Let’s ping our machine farm:

$ ansible -m ping machines tme-foo-m5 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python" }, "changed": false, "ping": "pong" } tme-foo-m4 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python" }, "changed": false, "ping": "pong" } tme-foo-m3 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python" }, "changed": false, "ping": "pong" } tme-foo-m2 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python" }, "changed": false, "ping": "pong" } tme-foo-m1 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python" }, "changed": false, "ping": "pong" } tme-foo-m6 | SUCCESS => { "ansible_facts": { "discovered_interpreter_python": "/usr/bin/python" }, "changed": false, "ping": "pong" }

As a good Linux citizen you always want to update to all the latest packages. I provided a crude package_update.yaml file for your convenience. It will also reboot the VMs once completed.

Important: The default password for the root user is still what that template shipped with. If you intend to use this for anything but a sandbox exercise, consider changing that root password.

Snapshot and Restore Virtual Machines

Now to the fun part. I’ve redacted a lot of the content I created for this project for many reasons, but it involved making customizations and installing proprietary software. In the various stages I wanted to create snapshots as some of these steps were not only lengthy, they were one-way streets. Creating a snapshot of the environment was indeed very handy.

To create a VM snapshot for the machines group:

$ ansible-playbook snapshot.yaml -e snapshot=goldenboy PLAY [localhost] **************************************************** TASK [Gathering Facts] ********************************************************************* Monday 04 November 2019 05:09:25 +0000 (0:00:00.096) 0:00:00.096 ******* ok: [localhost] TASK [snapshot : Create a VM snapshot] ********************************************************************* Monday 04 November 2019 05:09:27 +0000 (0:00:01.893) 0:00:01.989 ******* changed: [localhost -> localhost] => (item=tme-foo-m1) changed: [localhost -> localhost] => (item=tme-foo-m2) changed: [localhost -> localhost] => (item=tme-foo-m3) changed: [localhost -> localhost] => (item=tme-foo-m4) changed: [localhost -> localhost] => (item=tme-foo-m5) changed: [localhost -> localhost] => (item=tme-foo-m6) PLAY RECAP ********************************************************** localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 Monday 04 November 2019 05:09:35 +0000 (0:00:08.452) 0:00:10.442 ******* ===================================================================== snapshot : Create a VM snapshot ------------------------------- 8.45s Gathering Facts ----------------------------------------------- 1.89s Playbook run took 0 days, 0 hours, 0 minutes, 10 seconds

It’s now possible to trash the VM. If you ever want to go back:

$ ansible-playbook restore.yaml -e snapshot=goldenboy PLAY [localhost] **************************************************** TASK [Gathering Facts] ********************************************************************* Monday 04 November 2019 05:11:38 +0000 (0:00:00.104) 0:00:00.104 ******* ok: [localhost] TASK [restore : Revert a VM to a snapshot] ********************************************************************* Monday 04 November 2019 05:11:38 +0000 (0:00:00.860) 0:00:00.964 ******* changed: [localhost -> localhost] => (item=tme-foo-m1) changed: [localhost -> localhost] => (item=tme-foo-m2) changed: [localhost -> localhost] => (item=tme-foo-m3) changed: [localhost -> localhost] => (item=tme-foo-m4) changed: [localhost -> localhost] => (item=tme-foo-m5) changed: [localhost -> localhost] => (item=tme-foo-m6) TASK [restore : Power On VM] ********************************************************************* Monday 04 November 2019 05:11:47 +0000 (0:00:08.466) 0:00:09.431 ******* changed: [localhost -> localhost] => (item=tme-foo-m1) changed: [localhost -> localhost] => (item=tme-foo-m2) changed: [localhost -> localhost] => (item=tme-foo-m3) changed: [localhost -> localhost] => (item=tme-foo-m4) changed: [localhost -> localhost] => (item=tme-foo-m5) changed: [localhost -> localhost] => (item=tme-foo-m6) PLAY RECAP ********************************************************** localhost : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 Monday 04 November 2019 05:12:02 +0000 (0:00:15.232) 0:00:24.663 ******* ===================================================================== restore : Power On VM ---------------------------------------- 15.23s restore : Revert a VM to a snapshot --------------------------- 8.47s Gathering Facts ----------------------------------------------- 0.86s Playbook run took 0 days, 0 hours, 0 minutes, 24 seconds

Destroy Virtual Machines

I like things neat and tidy. This is how you would clean up after yourself:

$ ansible-playbook destroy.yaml PLAY [localhost] **************************************************** TASK [Gathering Facts] ********************************************************************* Monday 04 November 2019 05:13:12 +0000 (0:00:00.099) 0:00:00.099 ******* ok: [localhost] TASK [destroy : Destroy a virtual machine] ********************************************************************* Monday 04 November 2019 05:13:13 +0000 (0:00:00.870) 0:00:00.969 ******* changed: [localhost -> localhost] => (item=tme-foo-m1) changed: [localhost -> localhost] => (item=tme-foo-m2) changed: [localhost -> localhost] => (item=tme-foo-m3) changed: [localhost -> localhost] => (item=tme-foo-m4) changed: [localhost -> localhost] => (item=tme-foo-m5) changed: [localhost -> localhost] => (item=tme-foo-m6) PLAY RECAP ********************************************************** localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 Monday 04 November 2019 05:13:37 +0000 (0:00:24.141) 0:00:25.111 ******* ===================================================================== destroy : Destroy a virtual machine -------------------------- 24.14s Gathering Facts ----------------------------------------------- 0.87s Playbook run took 0 days, 0 hours, 0 minutes, 25 seconds

Summary

I probably dissed VMware more than necessary in this post. It’s a great infrastructure platform that is being deployed by 99% of the IT shops out there (don’t quote me on that). I hope you enjoyed this tutorial on how to make vSphere useful with Ansible.

Trivia: This tutorial brought you by one of the first few HPE Nimble Storage dHCI systems ever brought up!

[giosgdev]

Giosg automated deployment process almost removes all human error

[datamattsson]

Ansinimble Tutorial: Clone Refresh

Self servicing users are happy users. Assume we have a developer or QA engineer who wants to refresh a Nimble clone at will without getting too far into the weeds of becoming a fully fledged storage administrator. Ansinimble to the rescue!

This tutorial builds upon the Hello World tutorial posted a month ago. We’re going to assume there is a particular Nimble volume of interest already existing on the Nimble array that is subject to our clone refresh operation.

Setup

We’re going to assume the same basic setup, an Ansible host to run our playbooks on and a Ansible target host to refresh the clone on. This could be the same host.

Note: This tutorial will not go through some of the basic setup steps, follow the previous tutorial up to the "ping" steps before proceeding.

Start by checking out the git repo and cd into the clonerefresh directory:

git clone https://github.com/NimbleStorage/automation-examples cd automation-examples/ansinimble/clonerefresh

You’ll have the following directory structure:

. ├── ansible.cfg ├── group_vars │ └── prod │ └── main.yml ├── hosts ├── initial.yml └── refresh.yml

The first thing we need to go over is the inventory file:

[myhosts] sjc-tme-vlnx000 [mygroup] sjc-tme-nva000 ansible_connection=local [prod:children] myhosts mygroup

Edit the hostnames to correspond to your Linux host and Nimble array.

Next, we need to edit a few variables in group_vars/prod/main.yml:

--- nimble_group_options: ip-address: 10.18.160.197 username: admin nimble_group_password: admin nimble_linux_toolkit_bundle: /tmp/nlt_installer # My local vars my_volume_src: myvol1 # Clone refresh vars nimble_volume: mycln1 nimble_host_name: sjc-tme-vlnx000 nimble_volume_mountpoint: /mycln1

What needs attention here is the array details (see the first tutorial to get this right). Next we need to configure our source volume we want to clone from, in this case myvol1. The “Clone refresh vars” determines the clone name, which hostname the clone is going to be mapped to and where to mount it on the host.

Initial clone

If there’s already a clone setup on the host, you can skip this step.

For reference, here’s the initial clone setup playbook:

--- - hosts: mygroup roles: - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: snapshot, nimble_volume: "{{ my_volume_src }}" } - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: create, nimble_volume_from: "{{ my_volume_src }}" } - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: map, } - hosts: myhosts roles: - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: attach, } - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: mount, }

Very similar to our provision.yml file in the Hello World tutorial. Prior to creating the clone a snapshot is created (snapshot name is a timestamp) on the source volume and subsequently we clone from the source to a new volume. Maps, attaches and mounts.

Run with:

ansible-playbook -l "sjc-tme-nva000,sjc-tme-vlnx000" initial.yml

There should be a volume mounted on /mycln1, inspecting the array, it should indicate it’s a clone derived from the snapshot we created.

Refresh

The refresh operation autonomously umounts, detaches and destroys the clone, after that, creates a new snapshot, clones, attaches and mounts the new clone. The full playbook for reference:

--- - hosts: myhosts roles: - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: umount } - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: detach } - hosts: mygroup roles: - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: unmap } - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: delete } - hosts: mygroup roles: - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: snapshot, nimble_volume: "{{ my_volume_src }}" } - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: create, nimble_volume_from: "{{ my_volume_src }}" } - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: map, } - hosts: myhosts roles: - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: attach, } - { role: NimbleStorage.Ansinimble, nimble_object: volume, nimble_operation: mount, }

Refresh the clone with:

ansible-playbook -l "sjc-tme-nva000,sjc-tme-vlnx000" refresh.yml

The mount point should now contain new data from the source volume.

An example refresh would look like this (external site):

Security considerations

Letting a developer or QA engineer to have administrative privileges might not be desirable as they can essentially login to the array and wreak havoc. Wrapping the the execution of ansible-playbook can be done with various tools and abstractions but the most elegant solution is to set everyhing up in Ansible Tower. Rundeck also works or a sudo construct might be good enough.

Future

What is blatantly missing here is the deletion of the timestamped snapshots getting created on the source volume. This ability will be implemented in the future. Please follow this issue for updates. In the meantime manually delete snapshots from the UI or CLI.