Unattended upgrades in Debian

We use Debian for our systems. Not only because we love it, but also because you can harden and secure it very well. One of the many security precausions for Debian-based Linux distributions are unattended upgrades.

Unattended upgrades will (obviously) upgrade your Debian-based system unattendenly. If we keep it simple, this is basically an automated apt-get upgrade with some tweaks. That’s it – that’s unattended upgrades!

Why should I use them?

Well, it all depends on your environment and your patching behaviour and probably your (enterprise) processes. I had some discussions with friends and customers of ours, and this is what I’ve heard so far:

Pros for unattended upgrades

  • You want your systems always to be up-to-date
  • You want your systems always to have the latest security patches
  • You don’t want to patch them regularly by hand
  • Even if you miss the info of a new vulnerability, you want it to be patched

Cons for unattended upgrades

  • You can’t do it, because your enterprise company says you only have a few specific patch windows
  • You know your application breaks if you update it automatically
  • You want to do everything by hand, or at least invoke an automated upgrade process by hand, so you’ve everything under control
  • You’ve a proper vulnerability management and do security upgrades based on this

Never trust a human

I’m more the let’s-do-it-unattended-guy, because IMHO this is the much better option than the rest. I’ve also my issues with the reasons I’ve heard for the cons.

Let me start with the (in my opinion) most obvious problem in refusing unattended upgrades – the human being. We’re bad in doing things properly 100% of the time. We’re humans and this is part of our nature – we’re not perfect and we make mistakes, this is how we learn and evolve. But even more important, we might have bad days, we’re under stress, we’ve a lot of work, we go on vacation and we forget things. Now, what happens if there’s a new vulnerability and you’re just missed or forgot it? What happens if you’re on vacation, or you’ve way too much work to even check into the latest vulnerability? It probably won’t get patched (in time), and I think we all agree that this is really bad! So just having faith in a human operator is a bad decision 😉

Enterprise companies and their policies

As mentioned before, enterprise companies tend to define patch windows, which basically means you can only upgrade a certain server / service in a specific time window. That isn’t very agile, is it? You need to understand the motivation behind this management decisions. A lot of people, especially in management, are afraid of IT issues, such as downtimes. Because of that, they’re scared of changes, and they start to refuse them. Never touch a running system, right? 😉

But what if you can use that fear for your own advantage? What if you show them the downside of this decision? What if you use their fear against them? What if you say them something like:

So we’ve option A, which means we wait for your lovely patch window – which means our systems are vulnerable until that point – which means we can get hacked – which means that can harm the company – which means, of course, someone’s responsible for that little accident – which means you might think about it when you’re going to sleep.

Then we’ve option B, which means we do it automatically – which means our systems are always up-to-date – which means they’re secure – which means everybody’s happy – which means you can go to sleep with a clear conscience.

Just to be clear here, IMHO option A (in general) is the retarded option and only makes sense in some exceptions. You should definitly aim for option B.

Exceptions

Of course, not every environment is as black & white as described here. I just tried to keep it a bit witty. Still, you should try to avoid humans and do it automatically. Regardless of your decision, there might always be some exceptions. But that doesn’t mean you can’t do unattended upgrades for the majority of servers, services or packages.

Getting it up & ready

Installation

Debian already has a nice documentation about unattended upgrades. The documentation is nice and I used it to get it up & running. However, this is just for the basic setup and I had to figure out the deets by myself.

So let’s start by installing the required packages:

  • unattended-upgrades is doing the unattended upgrades (obvious, huh?)
  • apt-listchanges shows or mails the changes of a package before installing it

Configuration

The configuration files in question are the following:

First of all the 20auto-upgrades file:

This basically says do an apt-get update and apt-get upgrade every day, thus the number 1 (n-days) as value.

Next up, the 50unattended-upgrades file:

Now, in this file you’ve the choice which packages should be upgraded. Debian already explains a lot in the default configuration, which is:

Last but not least, the listchanges.conf file:

This is the file used to configure listchanges, which keeps track of changes. Just make sure your mail config is working and the email address is defined correctly.

Running it

unattended-ugprades is running automatically and is called via cronjob. You don’t need to install apticron or alike, as mentioned in several other posts.

If you want to debug it, you can easily run it on the CLI via:

Logging

If you want to know what was going on, check the logfile located here:

You can also add the following configuration to send you mails when an upgrade occured:

How we do it

Our configuration

For our infrastructure, we’re installing and configuring unattended-upgrades automatically via Ansible on every server. Here’s our default configuration with comments.

20auto-upgrades:

50unattended-upgrades:

Security-related vs. non-security-related updates

As you can see, we automatically install security-related updates every day in the background. This ensures our systems get the most important security patches. For the update of every other package, we use an Ansible playbook which simply does a dist-upgrade with a bit of exception handling, followed by running all configuration playbooks again with a final reboot.

Monitor for reboots

We do not automatically reboot our systems, because we don’t want them to go completely down in the middle of the day. Of course you can schedule it by defining the reboot time (other than now), but our approach is a bit different.

We simply monitor for the /var/run/reboot-required file in our monitoring and generate an alert in Zabbix:

Reboot alerts in Zabbix

This generates an alter to our Mattermost monitoring channel, so that everybody in our team knows about the “issue” and a reboot can be scheduled:

Verdict

Combining unattended security upgrades with automated attended full system upgrades is IMHO a nice way to keep your environment up-to-date and secure, without losing control of critical upgrades where a systems engineer is required.

2 Comments

  • ansibleftw Reply

    Hi, seeing your config file reminds me of Debops.
    So, do you use Debops to admin your infrastructure ?

    • Dominique Barton Reply

      Hi. I “know” Debops but nah, we’ve our own Ansible git project and doing configuring & deploying everything via our own Ansible roles. But nice 😉 seeing Debops people out there! 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.