ansible automation patching ubuntu

Ubuntu unattended updates with Ansible

luis.johnstone

If you’d like to enable unattended updates in Ubuntu and also configure a time for reboots to occur then the following should help you.
Now, manual steps can be found on the Ubuntu website but below I’ve drafted up some Ansible code for those of us that want to manage things using a Configuration as Code (CaS) model.

Our playbook consists of a small number of components:

  • Inventory file (usually found at: /etc/ansible/hosts )
  • Variables file (/vars/main.yml)
  • Playbook (unattended_upgrades.yml)
directory structure
.
├── unattended_upgrades.yml
├── vars
│   └── main.yml

Inventory

First off, you’ll want to build out your Ansible inventory so that you have three groups:

  • patch_group_a
  • patch_group_b
  • patch_group_c

For example:

/etc/ansible/hosts
[patch_group_a]
server1
server3

[patch_group_b]
server2
server4

[patch_group_c]
server5

Variables

The playbook assumes that you have a file under the ‘vars’ directory

You can change the group_names and timings by editing this file, which comes next:

vars/main.yml
---

reboot_offset: 15m

install_time: "{%- if 'patch_group_a' in group_names -%}
                  01:30
            {%- elif 'patch_group_b' in group_names -%}
                  02:30
            {%- elif 'patch_group_c' in group_names -%}
                  03:30
            {% else %}
                  03:30
            {%- endif -%}"

reboot_time: "{%- if 'patch_group_a' in group_names -%}
                  02:00
            {%- elif 'patch_group_b' in group_names -%}
                  03:00
            {%- elif 'patch_group_c' in group_names -%}
                  04:00
            {% else %}
                  04:00
            {%- endif -%}"

You can easily add or remove groups in the template or change the respective values for reboot_time.

Playbook

The main playbook does the following:

  • Installs: update-notifier-common
  • Creates: unattended upgrades configuration file
  • Enables: automated reboots

Note that the goal here is configuring an already existing service, namely, ‘unattended-upgrades‘. Specifically, we’re controlling the reboots.

What does the actual work of installing the updates is the unattended-upgrades script which gets invoked when the ‘apt-daily-upgrade.service‘ runs.

The timing for the execution of ‘apt-daily-upgrade.service‘ is controlled via a systemd timer named ‘apt-daily-upgrade.timer‘.

See the reference section at the end for full documentation.

Reboots

While this is sufficient to enable automatic reboots, it might not be exactly what you have in mind because the schedule by which the updates are installed is controlled separately via the apt-daily-upgrade.timer.

The configuration file can be found at:

/lib/systemd/system/apt-daily-upgrade.timer
[Unit]
Description=Daily apt upgrade and clean activities
After=apt-daily.timer

[Timer]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=60m
Persistent=true

[Install]
WantedBy=timers.target

As one can see there’s quite a bit of scope for customising when it will run and, ideally, you’ll probably want it to run as close to your specified reboot window as possible.
To achieve this using Ansible the vars file contains some jinja2 templating which allows us to dynamically set the value of variables based on whether a host is a member of an inventory group or not.

In our implementation here we’re reducing the randomized delay from 60 minutes down to 15 minutes, and setting a gap between install and reboot of 30 minutes. Given the offset value this means a worst-case scenario of a host having 15 minutes to install all updates before the reboot (and a best-case of 45 minutes). That’s fine for me but you may want to increase both values based on the needs of your environment.

Our final playbook will now look like this:

unattended_upgrades.yml
- name: Configure automated updates (Ubuntu)
  hosts: all
  become: true
  gather_facts: false
  vars_files:
    - vars/main.yml
  tasks:

## https://help.ubuntu.com/community/AutomaticSecurityUpdates

    - name: Install apt prerequisites for automated reboots
      ansible.builtin.apt:
        name: update-notifier-common
        state: present

    - name: Create unattended upgrades configuration file
      ansible.builtin.blockinfile:
        dest: /etc/apt/apt.conf.d/20auto-upgrades
        block: |
          APT::Periodic::Update-Package-Lists "1";
          APT::Periodic::Unattended-Upgrade "1";
        marker: "// {mark} ANSIBLE MANAGED BLOCK - unattended_upgrades settings"
        create: true
        mode: "0644"
        owner: root
        group: root
      register: unattended_upgrades_config_set

    - name: Enable automated reboots
      ansible.builtin.blockinfile:
        dest: /etc/apt/apt.conf.d/50unattended-upgrades
        block: |
          Unattended-Upgrade::Automatic-Reboot "true";
          Unattended-Upgrade::Automatic-Reboot-Time "{{ reboot_time }}";
        marker: "// {mark} ANSIBLE MANAGED BLOCK - unattended_upgrades settings"
        create: true
        mode: "0644"
        owner: root
        group: root
      register: unattended_upgrades_settings_set

    - name: Dpkg reconfigure
      ansible.builtin.command:
        cmd: dpkg-reconfigure -f noninteractive unattended-upgrades
      register: dpkg_reconfigure_unattended_upgrades
      when:
        - unattended_upgrades_config_set.changed or
          unattended_upgrades_settings_set.changed

    - name: Configure updates installation timing
      ansible.builtin.lineinfile:
        path: /lib/systemd/system/apt-daily-upgrade.timer
        regexp: '^OnCalendar'
        line: OnCalendar=*-*-* {{ install_time }}

    - name: Configure updates installation timing offset
      ansible.builtin.lineinfile:
        path: '/lib/systemd/system/apt-daily-upgrade.timer'
        regexp: '^RandomizedDelaySec'
        line: 'RandomizedDelaySec={{ reboot_offset}}'

Validation

Naturally, the best way to validate changes aside from checking the idempotency of the Ansible playbook is to see whether things work as expected. However, there are also some additional checks that you can run.

Firstly, you can verify whether the timer that triggers the service has picked up the correct schedule.

Let’s verify the config contents:

Bash
 grep -E OnCalendar /lib/systemd/system/apt-daily-upgrade.timer

Which should hopefully match what the playbook has configured. For example:

Bash
OnCalendar=*-*-*  01:30

Then we’ll check the actual value being used by the Systemd timer unit:

Bash
systemctl status apt-daily-upgrade.timer
● apt-daily-upgrade.timer - Daily apt upgrade and clean activities
     Loaded: loaded (/lib/systemd/system/apt-daily-upgrade.timer; enabled; preset: enabled)
     Active: active (waiting) since Fri 2023-10-20 22:22:53 BST; 17h ago
      Until: Fri 2023-10-20 22:22:53 BST; 17h ago
    Trigger: Sun 2023-10-22 01:37:11 BST; 9h left
   Triggers: ● apt-daily-upgrade.service

Note how the Trigger value matches up with our configured value, taking into account the randomized offset of up to 15 minutes (in this case our actual offset is about 8 minutes).

Now, if we travel forward in time to the date of the trigger and query the logs for the service:

Bash
grep -E apt-daily-upgrade.service /var/log/syslog

Whose output should show something like this:

Bash
2023-10-22T01:37:11.887415+01:00 localhost systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
2023-10-22T01:37:54.120174+01:00 localhost systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
2023-10-22T01:37:54.121916+01:00 localhost systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
2023-10-22T01:37:54.123701+01:00 localhost systemd[1]: apt-daily-upgrade.service: Consumed 9.982s CPU time.

Another approach for validating the remaining settings is to use the apt-config dump command:

Bash
apt-config dump APT::Periodic::Unattended-Upgrade && \
apt-config dump APT::Periodic::Update-Package-Lists && \
apt-config dump Unattended-Upgrade::Automatic-Reboot && \
apt-config dump Unattended-Upgrade::Automatic-Reboot-Time

which should hopefully give us the results that we expect:

Bash
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::Update-Package-Lists "1";
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "02:00";

A note on update types

By default the unattended-upgrades package is configured to install only security updates. However, you can easily change this by uncommenting what you want from the configuration file. Here’s the top of the file and the settings that you want:

/etc/apt/apt.conf.d/50unattended-upgrades
// Automatically upgrade packages from these (origin:archive) pairs
//
// Note that in Ubuntu security updates may pull in new dependencies
// from non-security sources (e.g. chromium). By allowing the release
// pocket these get automatically pulled in.
Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}";
        "${distro_id}:${distro_codename}-security";
        // Extended Security Maintenance; doesn't necessarily exist for
        // every release and this system may not have it installed, but if
        // available, the policy for updates is such that unattended-upgrades
        // should also install from here by default.
        "${distro_id}ESMApps:${distro_codename}-apps-security";
        "${distro_id}ESM:${distro_codename}-infra-security";
//      "${distro_id}:${distro_codename}-updates";
//      "${distro_id}:${distro_codename}-proposed";
//      "${distro_id}:${distro_codename}-backports";
};

If you want to update applications in the same way as if you’d run an apt-upgrade then you’ll probably want to uncomment the line highlighted above.

Special considerations and issues

  • Please note that the comment-syntax for the Ubuntu unattended config files uses double-forward slashes (e.g. ‘//’) and therefore using hashes for comments will cause a failure (e.g. ‘#’). As a result, the markers for the Ansible Blockinfile module must use the double-forward slash method.
  • Since this all relies on unattended-upgrades functioning you’ll want to make sure everything is already working by checking your logs under:
logfiles location
/var/log/unattended-upgrades/

References

https://help.ubuntu.com/community/AutomaticSecurityUpdates
https://manpages.ubuntu.com/manpages/lunar/en/man8/unattended-upgrade.8.html

Leave a Reply

Your email address will not be published. Required fields are marked *