Problem: I have several systems with various operating systems within the starlabs (Brian's home lab) network that I need to keep up to date.
Solution: I automated this process so that I can update all of them from the control plane node.
On Control Plane Node: Be sure you copy the config and enable logging
root@ironman6:~# mv /etc/ansible/ansible.cfg /etc/ansible/ansible-default.cfg.bak root@ironman6:~# cp -a /etc/ansible/ansible-sample.cfg /etc/ansible/ansible.cfg root@ironman6:~# grep log_path /etc/ansible/ansible.cfg ;log_path= root@ironman6:~# vim /etc/ansible/ansible.cfg root@ironman6:~# grep log_path /etc/ansible/ansible.cfg log_path=/var/log/ansible.log root@ironman6:~# touch /var/log/ansible.log
On Control Plane Node: I added my systems to my inventory here, at the bottom of the file
/etc/ansible/hosts
(truncated output) [starlabs] veronica ansible_host=172.16.0.8 ansible_port=22 ansible_user=ansible ironman6 ansible_host=127.0.0.1 ironman9 ansible_host=172.16.0.8 ansible_port=2222 tardis ansible_host=172.16.0.17 [all:vars] ansible_python_interpreter=/usr/bin/python3
On Control Plane Node: Here is my update inventory with run counts.
root@ironman6:~# history | grep -v grep | grep -v "#" | grep ansible-playbook\ \-v | grep update | awk '{print $2,$3,$4,$5,$6,$7}' | sort | uniq -c | sort -rnk1 | head -5
33 ansible-playbook -v /playbooks/update-veronica.starlabs.us.yml --become -u ansible
22 ansible-playbook -v /playbooks/update-ironman6.starlabs.us.yml --become -u ansible
17 ansible-playbook -v /playbooks/update-ironman9.starlabs.us.yml --become -u ansible
14 ansible-playbook -v /playbooks/update-tardis.starlabs.us.yml --become -u ansible
On Server: Add the ansible user
root@ironman6:~# adduser ansible
Adding user `ansible' ...
Adding new group `ansible' (1006) ...
Adding new user `ansible' (1006) with group `ansible' ...
Creating home directory `/home/ansible' ...
Copying files from `/etc/skel' ...
New password:
Retype new password:
passwd: password updated successfully
Changing the user information for ansible
Enter the new value, or press ENTER for the default
Full Name []:
Room Number []:
Work Phone []:
Home Phone []:
Other []:
Is the information correct? [Y/n] y
root@ironman6:~# usermod -aG sudo ansible
On Server: Make the ssh key
Note: For extra security, set a passphrase. But then you'll have to enter that passphrase everytime you run ansible. I prefer to just be sure my control plane node never ever gets compromised.
root@ironman6:~# su - ansible ansible@ironman6:~$ mkdir .ssh ansible@ironman6:~$ ssh-keygen -t rsa -b 4096 -f "/home/ansible"/.ssh/ssh_key.key -C "starlabs-$(date -u +%Y-%m-%d-%H:%M:%S%z)"; v_KEY_CREATED=true Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/ansible/.ssh/ssh_key.key Your public key has been saved in /home/ansible/.ssh/ssh_key.key.pub The key fingerprint is: SHA256:GT59w4qz0K6Hx9gWZJKVTGsOERdQvNXE4VzuRXOJPtk starlabs-2023-06-05-05:42:49+0000 The key's randomart image is: +---[RSA 4096]----+ | +B=o +o.o.+| | o=..oo+ oo| | .o+o + + .| | o=++ . = E | | +S . + o | | ..o o . | | .=+.. | | oo*o | | .=o | +----[SHA256]-----+
On Server: Authorize the key
ansible@ironman6:~$ touch ~/.ssh/authorized_keys ansible@ironman6:~$ chmod 600 ~/.ssh/authorized_keys ansible@ironman6:~$ cat /home/ansible/.ssh/ssh_key.key.pub > /home/ansible/.ssh/authorized_keys
On Server: Get the private key
ansible@ironman6:~$ cat /home/ansible/.ssh/ssh_key.key -----BEGIN OPENSSH PRIVATE KEY----- hidden output -----END OPENSSH PRIVATE KEY-----
On Control Plane Node: Copy the private key to the control plane node
root@ironman6:~# vim ~/.ssh/ironman6.starlabs.us.key root@ironman6:~# chmod 600 ~/.ssh/ironman6.starlabs.us.key
On Control Plane Node: Test out the private key
root@ironman6:~# ssh ansible@ironman6.starlabs.us -i /root/.ssh/ironman6.starlabs.us.key -p22 Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-1107-raspi aarch64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/pro New release '22.04.3 LTS' available. Run 'do-release-upgrade' to upgrade to it. Last login: Fri May 3 10:18:17 2024 from 127.0.0.1 ansible@ironman6:~$
On Control Plane Node: Test remotely running a command on the server.
root@ironman6:~# SERVER=ironman6;ansible $SERVER -m shell -a 'cat /etc/os-release' -u ansible --key-file ~/.ssh/$SERVER.starlabs.us.key ironman6 | CHANGED | rc=0 >> NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal
On Control Plane Node: Add the update playbook
Note: You can add multiple hosts in the "hosts:" spot, seperated by spaces.
/playbooks/update-ironman6.starlabs.us.yml
---
- hosts: ironman6
vars:
ansible_ssh_private_key_file: ~/.ssh/ironman6.starlabs.us.key
tasks:
- name: Update and upgrade apt packages
become: true
apt:
upgrade: yes
update_cache: yes
#cache_valid_time: 86400 #One day
- name: Update snap packages
become: true
command: snap refresh
On Server: Remove the sudo password from the ansible user by adding to the bottom of /etc/sudoers
ansible ALL=(ALL) NOPASSWD: ALL
It should be below the #includedir /etc/sudoers.d part
You can use visudo to do this
root@ironman6:~# visudo # See sudoers(5) for more information on "#include" directives: #includedir /etc/sudoers.d ansible ALL=(ALL) NOPASSWD: ALL ^G Get Help ^O Write Out ^W Where Is ^K Cut Text ^J Justify ^C Cur Pos M-U Undo M-A Mark Text M-] To Bracket M-Q Previous ^X Exit ^R Read File ^\ Replace ^U Paste Text ^T To Spell ^_ Go To Line M-E Redo M-6 Copy Text ^Q Where Was M-W Next root@ironman6:~# tail -2 /etc/sudoers #includedir /etc/sudoers.d ansible ALL=(ALL) NOPASSWD: ALL
On Control Plane Node: Remotely run updates for the server.
root@ironman6:~# ansible-playbook -v /playbooks/update-ironman6.starlabs.us.yml --become -u ansible
Using /etc/ansible/ansible.cfg as config file
PLAY [ironman6] ******************************************************************************************************************************
TASK [Gathering Facts] ***********************************************************************************************************************
ok: [ironman6]
TASK [Update and upgrade apt packages] *******************************************************************************************************
changed: [ironman6] => {"changed": true, "msg": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nCalculat
(truncated output)
TASK [Update snap packages] ******************************************************************************************************************
changed: [ironman6] => {"changed": true, "cmd": ["snap", "refresh"], "delta": "0:01:09.182339", "end": "2024-05-25 13:53:42.117539", "msg": ""
PLAY RECAP ***********************************************************************************************************************************
ironman6 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
On Server: Add the ansible user
root@veronica:~# adduser ansible
Adding user `ansible' ...
Adding new group `ansible' (1006) ...
Adding new user `ansible' (1006) with group `ansible' ...
Creating home directory `/home/ansible' ...
Copying files from `/etc/skel' ...
New password:
Retype new password:
passwd: password updated successfully
Changing the user information for ansible
Enter the new value, or press ENTER for the default
Full Name []:
Room Number []:
Work Phone []:
Home Phone []:
Other []:
Is the information correct? [Y/n] y
root@veronica:~# usermod -aG sudo ansible
On Server: Make the ssh key
Note: For extra security, set a passphrase. But then you'll have to enter that passphrase everytime you run ansible. I prefer to just be sure my control plane node never ever gets compromised.
root@veronica:~# su - ansible ansible@veronica:~$ mkdir .ssh ansible@veronica:~$ ssh-keygen -t rsa -b 4096 -f "/home/ansible"/.ssh/ssh_key.key -C "starlabs-$(date -u +%Y-%m-%d-%H:%M:%S%z)"; v_KEY_CREATED=true Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/ansible/.ssh/ssh_key.key Your public key has been saved in /home/ansible/.ssh/ssh_key.key.pub The key fingerprint is: SHA256:GT59w4qz0K6Hx9gWZJKVTGsOERdQvNXE4VzuRXOJPtk starlabs-2023-06-05-05:42:49+0000 The key's randomart image is: +---[RSA 4096]----+ | +B=o +o.o.+| | o=..oo+ oo| | .o+o + + .| | o=++ . = E | | +S . + o | | ..o o . | | .=+.. | | oo*o | | .=o | +----[SHA256]-----+
On Server: Authorize the key
ansible@veronica:~$ touch ~/.ssh/authorized_keys ansible@veronica:~$ chmod 600 ~/.ssh/authorized_keys ansible@veronica:~$ cat /home/ansible/.ssh/ssh_key.key.pub > /home/ansible/.ssh/authorized_keys
On Server: Get the private key
ansible@veronica:~$ cat /home/ansible/.ssh/ssh_key.key -----BEGIN OPENSSH PRIVATE KEY----- hidden output -----END OPENSSH PRIVATE KEY-----
On Control Plane Node: Copy the private key to the control plane node
root@ironman6:~# vim ~/.ssh/veronica.starlabs.us.key root@ironman6:~# chmod 600 ~/.ssh/veronica.starlabs.us.key
On Control Plane Node: Test out the private key
root@ironman6:~# ssh ansible@veronica.starlabs.us -i /root/.ssh/veronica.starlabs.us.key -p22 Linux veronica.starlabs.us 4.19.0-26-amd64 #1 SMP Debian 4.19.304-1 (2024-01-09) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Fri May 10 20:38:38 2024 from 172.16.0.13 ansible@veronica:~$
On Control Plane Node: Test remotely running a command on the server.
root@ironman6:~# SERVER=veronica;ansible $SERVER -m shell -a 'cat /etc/os-release' -u ansible --key-file ~/.ssh/$SERVER.starlabs.us.key veronica | CHANGED | rc=0 >> PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
On Control Plane Node: Add the update playbook
Note: You can add multiple hosts in the "hosts:" spot, seperated by spaces.
/playbooks/update-veronica.starlabs.us.yml
---
- hosts: veronica
vars:
ansible_ssh_private_key_file: ~/.ssh/veronica.starlabs.us.key
tasks:
- name: Update and upgrade apt packages
become: true
apt:
upgrade: yes
update_cache: yes
#cache_valid_time: 86400 #One day
# - name: Update snap packages
# become: true
# command: snap refresh
On Server: Remove the sudo password from the ansible user by adding to the bottom of /etc/sudoers
ansible ALL=(ALL) NOPASSWD: ALL
It should be below the #includedir /etc/sudoers.d part
You can use visudo to do this
root@veronica:~# visudo # See sudoers(5) for more information on "#include" directives: #includedir /etc/sudoers.d ansible ALL=(ALL) NOPASSWD: ALL ^G Get Help ^O Write Out ^W Where Is ^K Cut Text ^J Justify ^C Cur Pos M-U Undo M-A Mark Text M-] To Bracket M-Q Previous ^X Exit ^R Read File ^\ Replace ^U Paste Text ^T To Spell ^_ Go To Line M-E Redo M-6 Copy Text ^Q Where Was M-W Next root@veronica:~# tail -2 /etc/sudoers #includedir /etc/sudoers.d ansible ALL=(ALL) NOPASSWD: ALL
On Control Plane Node: Remotely run updates for the server.
root@ironman6:~# ansible-playbook -v /playbooks/update-veronica.starlabs.us.yml --become -u ansible
Using /etc/ansible/ansible.cfg as config file
PLAY [veronica] ******************************************************************************************************************************
TASK [Gathering Facts] ***********************************************************************************************************************
ok: [veronica]
TASK [Update and upgrade apt packages] *******************************************************************************************************
changed: [veronica] => {"changed": true, "msg": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nCalculat
(truncated output)
TASK [Update snap packages] ******************************************************************************************************************
changed: [veronica] => {"changed": true, "cmd": ["snap", "refresh"], "delta": "0:01:09.182339", "end": "2024-05-25 13:53:42.117539", "msg": ""
PLAY RECAP ***********************************************************************************************************************************
veronica : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
On Server: Add the ansible user
[root@ironman9 ~]# adduser ansible [root@ironman9 ~]# usermod -aG wheel ansible
On Server: Make the ssh key
Note: For extra security, set a passphrase. But then you'll have to enter that passphrase everytime you run ansible. I prefer to just be sure my control plane node never ever gets compromised.
[root@ironman9 ~]# su - ansible [ansible@ironman9 ~]$ mkdir .ssh [ansible@ironman9 ~]$ ssh-keygen -t rsa -b 4096 -f "/home/ansible"/.ssh/ssh_key.key -C "starlabs-$(date -u +%Y-%m-%d-%H:%M:%S%z)"; v_KEY_CREATED=true Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/ansible/.ssh/ssh_key.key Your public key has been saved in /home/ansible/.ssh/ssh_key.key.pub The key fingerprint is: SHA256:GT59w4qz0K6Hx9gWZJKVTGsOERdQvNXE4VzuRXOJPtk starlabs-2023-06-05-05:42:49+0000 The key's randomart image is: +---[RSA 4096]----+ | +B=o +o.o.+| | o=..oo+ oo| | .o+o + + .| | o=++ . = E | | +S . + o | | ..o o . | | .=+.. | | oo*o | | .=o | +----[SHA256]-----+
On Server: Authorize the key
[ansible@ironman9 ~]$ touch ~/.ssh/authorized_keys [ansible@ironman9 ~]$ chmod 600 ~/.ssh/authorized_keys [ansible@ironman9 ~]$ cat /home/ansible/.ssh/ssh_key.key.pub > /home/ansible/.ssh/authorized_keys
On Server: Get the private key
[ansible@ironman9 ~]$ cat /home/ansible/.ssh/ssh_key.key -----BEGIN OPENSSH PRIVATE KEY----- hidden output -----END OPENSSH PRIVATE KEY-----
On Control Plane Node: Copy the private key to the control plane node
root@ironman6:~# vim ~/.ssh/ironman9.starlabs.us.key root@ironman6:~# chmod 600 ~/.ssh/ironman9.starlabs.us.key
On Control Plane Node: Test out the private key
root@ironman6:~# ssh ansible@ironman9.starlabs.us -i /root/.ssh/ironman9.starlabs.us.key -p2222 Last login: Mon Jan 1 08:38:57 2024 from 172.16.0.13 [ansible@ironman9 ~]$
On Control Plane Node: Test remotely running a command on the server.
root@ironman6:~# SERVER=ironman9;ansible $SERVER -m shell -a 'cat /etc/os-release' -u ansible --key-file ~/.ssh/$SERVER.starlabs.us.key ironman9 | CHANGED | rc=0 >> NAME="AlmaLinux" VERSION="8.8 (Sapphire Caracal)" ID="almalinux" ID_LIKE="rhel centos fedora" VERSION_ID="8.8" PLATFORM_ID="platform:el8" PRETTY_NAME="AlmaLinux 8.8 (Sapphire Caracal)" ANSI_COLOR="0;34" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:almalinux:almalinux:8::baseos" HOME_URL="https://almalinux.org/" DOCUMENTATION_URL="https://wiki.almalinux.org/" BUG_REPORT_URL="https://bugs.almalinux.org/" ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8" ALMALINUX_MANTISBT_PROJECT_VERSION="8.8" REDHAT_SUPPORT_PRODUCT="AlmaLinux" REDHAT_SUPPORT_PRODUCT_VERSION="8.8"
On Control Plane Node: Add the update playbook
Note: You can add multiple hosts in the "hosts:" spot, seperated by spaces.
/playbooks/update-ironman9.starlabs.us.yml
---
- hosts: ironman9
vars:
ansible_ssh_private_key_file: ~/.ssh/ironman9.starlabs.us.key
tasks:
- name: Update all packages
become: true
dnf:
state: latest
# - name: Restart services
# service:
# state: restarted
# name: "{{ item }}"
# loop:
# - httpd
# - mysql
# - name: Update snap packages
# become: true
# command: snap refresh
On Server: Remove the sudo password from the ansible user by adding to the bottom of /etc/sudoers
ansible ALL=(ALL) NOPASSWD: ALL
It should be below the #includedir /etc/sudoers.d part
You can use visudo to do this
root@ironman9:~# visudo ## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment) #includedir /etc/sudoers.d ansible ALL=(ALL) NOPASSWD: ALL root@ironman9:~# tail -2 /etc/sudoers #includedir /etc/sudoers.d ansible ALL=(ALL) NOPASSWD: ALL
On Control Plane Node: Remotely run updates for the server.
[root@ironman9 ~]# ansible-playbook -v /playbooks/update-ironman9.starlabs.us.yml --become -u ansible
Using /etc/ansible/ansible.cfg as config file
PLAY [ironman9] ******************************************************************************************************************************
TASK [Gathering Facts] ***********************************************************************************************************************
ok: [ironman9]
TASK [Update and upgrade apt packages] *******************************************************************************************************
changed: [ironman9] => {"changed": true, "msg": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nCalculat
(truncated output)
TASK [Update snap packages] ******************************************************************************************************************
changed: [ironman9] => {"changed": true, "cmd": ["snap", "refresh"], "delta": "0:01:09.182339", "end": "2024-05-25 13:53:42.117539", "msg": ""
PLAY RECAP ***********************************************************************************************************************************
ironman9 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0