We use Debian for our systems. Not only because we love it, but also because you can harden and secure it very well. One of the many security precausions for Debian-based Linux distributions are unattended upgrades.
Unattended upgrades will (obviously) upgrade your Debian-based system unattendenly. If we keep it simple, this is basically an automated apt-get upgrade with some tweaks. That’s it – that’s unattended upgrades!
Why should I use them?
Well, it all depends on your environment and your patching behaviour and probably your (enterprise) processes. I had some discussions with friends and customers of ours, and this is what I’ve heard so far:
Pros for unattended upgrades
- You want your systems always to be up-to-date
- You want your systems always to have the latest security patches
- You don’t want to patch them regularly by hand
- Even if you miss the info of a new vulnerability, you want it to be patched
Cons for unattended upgrades
- You can’t do it, because your enterprise company says you only have a few specific patch windows
- You know your application breaks if you update it automatically
- You want to do everything by hand, or at least invoke an automated upgrade process by hand, so you’ve everything under control
- You’ve a proper vulnerability management and do security upgrades based on this
Never trust a human
I’m more the let’s-do-it-unattended-guy, because IMHO this is the much better option than the rest. I’ve also my issues with the reasons I’ve heard for the cons.
Let me start with the (in my opinion) most obvious problem in refusing unattended upgrades – the human being. We’re bad in doing things properly 100% of the time. We’re humans and this is part of our nature – we’re not perfect and we make mistakes, this is how we learn and evolve. But even more important, we might have bad days, we’re under stress, we’ve a lot of work, we go on vacation and we forget things. Now, what happens if there’s a new vulnerability and you’re just missed or forgot it? What happens if you’re on vacation, or you’ve way too much work to even check into the latest vulnerability? It probably won’t get patched (in time), and I think we all agree that this is really bad! So just having faith in a human operator is a bad decision 😉
Enterprise companies and their policies
As mentioned before, enterprise companies tend to define patch windows, which basically means you can only upgrade a certain server / service in a specific time window. That isn’t very agile, is it? You need to understand the motivation behind this management decisions. A lot of people, especially in management, are afraid of IT issues, such as downtimes. Because of that, they’re scared of changes, and they start to refuse them. Never touch a running system, right? 😉
But what if you can use that fear for your own advantage? What if you show them the downside of this decision? What if you use their fear against them? What if you say them something like:
So we’ve option A, which means we wait for your lovely patch window – which means our systems are vulnerable until that point – which means we can get hacked – which means that can harm the company – which means, of course, someone’s responsible for that little accident – which means you might think about it when you’re going to sleep.
Then we’ve option B, which means we do it automatically – which means our systems are always up-to-date – which means they’re secure – which means everybody’s happy – which means you can go to sleep with a clear conscience.
Just to be clear here, IMHO option A (in general) is the retarded option and only makes sense in some exceptions. You should definitly aim for option B.
Exceptions
Of course, not every environment is as black & white as described here. I just tried to keep it a bit witty. Still, you should try to avoid humans and do it automatically. Regardless of your decision, there might always be some exceptions. But that doesn’t mean you can’t do unattended upgrades for the majority of servers, services or packages.
Getting it up & ready
Installation
Debian already has a nice documentation about unattended upgrades. The documentation is nice and I used it to get it up & running. However, this is just for the basic setup and I had to figure out the deets by myself.
So let’s start by installing the required packages:
apt-get install unattended-upgrades apt-listchanges
- unattended-upgrades is doing the unattended upgrades (obvious, huh?)
- apt-listchanges shows or mails the changes of a package before installing it
Configuration
The configuration files in question are the following:
/etc/apt/apt.conf.d/20auto-upgrades /etc/apt/apt.conf.d/50unattended-upgrades /etc/apt/listchanges.conf
First of all the 20auto-upgrades file:
APT::Periodic::Update-Package-Lists "1"; APT::Periodic::Unattended-Upgrade "1";
This basically says do an apt-get update and apt-get upgrade every day, thus the number 1 (n-days) as value.
Next up, the 50unattended-upgrades file:
Unattended-Upgrade::Origins-Pattern {
"origin=Debian,codename=${distro_codename},label=Debian-Security";
};
Unattended-Upgrade::Package-Blacklist {
};
Now, in this file you’ve the choice which packages should be upgraded. Debian already explains a lot in the default configuration, which is:
// Lines below have the format format is "keyword=value,...". A
// package will be upgraded only if the values in its metadata match
// all the supplied keywords in a line. (In other words, omitted
// keywords are wild cards.) The keywords originate from the Release
// file, but several aliases are accepted. The accepted keywords are:
// a,archive,suite (eg, "stable")
// c,component (eg, "main", "contrib", "non-free")
// l,label (eg, "Debian", "Debian-Security")
// o,origin (eg, "Debian", "Unofficial Multimedia Packages")
// n,codename (eg, "jessie", "jessie-updates")
// site (eg, "http.debian.net")
// The available values on the system are printed by the command
// "apt-cache policy", and can be debugged by running
// "unattended-upgrades -d" and looking at the log file.
//
// Within lines unattended-upgrades allows 2 macros whose values are
// derived from /etc/debian_version:
// ${distro_id} Installed origin.
// ${distro_codename} Installed codename (eg, "jessie")
Last but not least, the listchanges.conf file:
[apt] frontend=pager confirm=false email_address=root save_seen=/var/lib/apt/listchanges.db which=news
This is the file used to configure listchanges, which keeps track of changes. Just make sure your mail config is working and the email address is defined correctly.
Running it
unattended-ugprades is running automatically and is called via cronjob. You don’t need to install apticron or alike, as mentioned in several other posts.
If you want to debug it, you can easily run it on the CLI via:
unattended-upgrades -d
Logging
If you want to know what was going on, check the logfile located here:
/var/log/unattended-upgrades/
You can also add the following configuration to send you mails when an upgrade occured:
Unattended-Upgrade::Mail "root"; Unattended-Upgrade::MailOnlyOnError "false";
How we do it
Our configuration
For our infrastructure, we’re installing and configuring unattended-upgrades automatically via Ansible on every server. Here’s our default configuration with comments.
20auto-upgrades:
// // !!! WARNING: This file is managed by Ansible, DON'T EDIT it manually !!! // // Enable unattended upgrades. APT::Periodic::Enable "1"; // Do "apt-get upgrade" every n-days (0=disable). APT::Periodic::Unattended-Upgrade "1"; // Do "apt-get upgrade --download-only" every n-days (0=disable). APT::Periodic::Update-Package-Lists "1"; // Do "apt-get upgrade --download-only" every n-days (0=disable). APT::Periodic::Download-Upgradeable-Packages "1"; // Do "apt-get autoclean" every n-days (0=disable). APT::Periodic::AutocleanInterval "7";
50unattended-upgrades:
//
// !!! WARNING: This file is managed by Ansible, DON'T EDIT it manually !!!
//
Unattended-Upgrade::Origins-Pattern {
"o=Debian,n=${distro_codename},l=Debian-Security";
};
Unattended-Upgrade::Package-Blacklist {
};
// Automatically run "dpkg --force-confold --configure -a".
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
// Do automatic removal of new unused dependencies after the upgrade.
Unattended-Upgrade::Remove-Unused-Dependencies "true";
// Install upgrades when the machine is shuting down instead of doing it in the background.
Unattended-Upgrade::InstallOnShutdown "false";
// Automatically reboot if the file /var/run/reboot-required is found after the upgrade.
Unattended-Upgrade::Automatic-Reboot "false";
// Automatically reboot even if there are users currently logged in.
Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
// If automatic reboot is enabled and needed, reboot at the specific time.
Unattended-Upgrade::Automatic-Reboot-Time "now";
// Enable mail notifications.
Unattended-Upgrade::Mail "<our-own-mail>@confirm.ch";
Unattended-Upgrade::MailOnlyOnError "true";
// Enable logging via syslog.
Unattended-Upgrade::SyslogEnable "true";
Unattended-Upgrade::SyslogFacility "daemon";
Security-related vs. non-security-related updates
As you can see, we automatically install security-related updates every day in the background. This ensures our systems get the most important security patches. For the update of every other package, we use an Ansible playbook which simply does a dist-upgrade with a bit of exception handling, followed by running all configuration playbooks again with a final reboot.
Monitor for reboots
We do not automatically reboot our systems, because we don’t want them to go completely down in the middle of the day. Of course you can schedule it by defining the reboot time (other than now), but our approach is a bit different.
We simply monitor for the /var/run/reboot-required file in our monitoring and generate an alert in Zabbix:
This generates an alter to our Mattermost monitoring channel, so that everybody in our team knows about the “issue” and a reboot can be scheduled:
Verdict
Combining unattended security upgrades with automated attended full system upgrades is IMHO a nice way to keep your environment up-to-date and secure, without losing control of critical upgrades where a systems engineer is required.
76 Comments