Unattended upgrades in Debian

We use Debian for our systems. Not only because we love it, but also because you can harden and secure it very well. One of the many security precausions for Debian-based Linux distributions are unattended upgrades.

Unattended upgrades will (obviously) upgrade your Debian-based system unattendenly. If we keep it simple, this is basically an automated apt-get upgrade with some tweaks. That’s it – that’s unattended upgrades!

Why should I use them?

Well, it all depends on your environment and your patching behaviour and probably your (enterprise) processes. I had some discussions with friends and customers of ours, and this is what I’ve heard so far:

Pros for unattended upgrades

  • You want your systems always to be up-to-date
  • You want your systems always to have the latest security patches
  • You don’t want to patch them regularly by hand
  • Even if you miss the info of a new vulnerability, you want it to be patched

Cons for unattended upgrades

  • You can’t do it, because your enterprise company says you only have a few specific patch windows
  • You know your application breaks if you update it automatically
  • You want to do everything by hand, or at least invoke an automated upgrade process by hand, so you’ve everything under control
  • You’ve a proper vulnerability management and do security upgrades based on this

Never trust a human

I’m more the let’s-do-it-unattended-guy, because IMHO this is the much better option than the rest. I’ve also my issues with the reasons I’ve heard for the cons.

Let me start with the (in my opinion) most obvious problem in refusing unattended upgrades – the human being. We’re bad in doing things properly 100% of the time. We’re humans and this is part of our nature – we’re not perfect and we make mistakes, this is how we learn and evolve. But even more important, we might have bad days, we’re under stress, we’ve a lot of work, we go on vacation and we forget things. Now, what happens if there’s a new vulnerability and you’re just missed or forgot it? What happens if you’re on vacation, or you’ve way too much work to even check into the latest vulnerability? It probably won’t get patched (in time), and I think we all agree that this is really bad! So just having faith in a human operator is a bad decision 😉

Enterprise companies and their policies

As mentioned before, enterprise companies tend to define patch windows, which basically means you can only upgrade a certain server / service in a specific time window. That isn’t very agile, is it? You need to understand the motivation behind this management decisions. A lot of people, especially in management, are afraid of IT issues, such as downtimes. Because of that, they’re scared of changes, and they start to refuse them. Never touch a running system, right? 😉

But what if you can use that fear for your own advantage? What if you show them the downside of this decision? What if you use their fear against them? What if you say them something like:

So we’ve option A, which means we wait for your lovely patch window – which means our systems are vulnerable until that point – which means we can get hacked – which means that can harm the company – which means, of course, someone’s responsible for that little accident – which means you might think about it when you’re going to sleep.

Then we’ve option B, which means we do it automatically – which means our systems are always up-to-date – which means they’re secure – which means everybody’s happy – which means you can go to sleep with a clear conscience.

Just to be clear here, IMHO option A (in general) is the retarded option and only makes sense in some exceptions. You should definitly aim for option B.

Exceptions

Of course, not every environment is as black & white as described here. I just tried to keep it a bit witty. Still, you should try to avoid humans and do it automatically. Regardless of your decision, there might always be some exceptions. But that doesn’t mean you can’t do unattended upgrades for the majority of servers, services or packages.

Getting it up & ready

Installation

Debian already has a nice documentation about unattended upgrades. The documentation is nice and I used it to get it up & running. However, this is just for the basic setup and I had to figure out the deets by myself.

So let’s start by installing the required packages:

apt-get install unattended-upgrades apt-listchanges
  • unattended-upgrades is doing the unattended upgrades (obvious, huh?)
  • apt-listchanges shows or mails the changes of a package before installing it

Configuration

The configuration files in question are the following:

/etc/apt/apt.conf.d/20auto-upgrades
/etc/apt/apt.conf.d/50unattended-upgrades
/etc/apt/listchanges.conf

First of all the 20auto-upgrades file:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

This basically says do an apt-get update and apt-get upgrade every day, thus the number 1 (n-days) as value.

Next up, the 50unattended-upgrades file:

Unattended-Upgrade::Origins-Pattern {
        "origin=Debian,codename=${distro_codename},label=Debian-Security";
};

Unattended-Upgrade::Package-Blacklist {
};

Now, in this file you’ve the choice which packages should be upgraded. Debian already explains a lot in the default configuration, which is:

// Lines below have the format format is "keyword=value,...".  A
// package will be upgraded only if the values in its metadata match
// all the supplied keywords in a line.  (In other words, omitted
// keywords are wild cards.) The keywords originate from the Release
// file, but several aliases are accepted.  The accepted keywords are:
//   a,archive,suite (eg, "stable")
//   c,component     (eg, "main", "contrib", "non-free")
//   l,label         (eg, "Debian", "Debian-Security")
//   o,origin        (eg, "Debian", "Unofficial Multimedia Packages")
//   n,codename      (eg, "jessie", "jessie-updates")
//     site          (eg, "http.debian.net")
// The available values on the system are printed by the command
// "apt-cache policy", and can be debugged by running
// "unattended-upgrades -d" and looking at the log file.
//
// Within lines unattended-upgrades allows 2 macros whose values are
// derived from /etc/debian_version:
//   ${distro_id}            Installed origin.
//   ${distro_codename}      Installed codename (eg, "jessie")

Last but not least, the listchanges.conf file:

[apt]
frontend=pager
confirm=false
email_address=root
save_seen=/var/lib/apt/listchanges.db
which=news

This is the file used to configure listchanges, which keeps track of changes. Just make sure your mail config is working and the email address is defined correctly.

Running it

unattended-ugprades is running automatically and is called via cronjob. You don’t need to install apticron or alike, as mentioned in several other posts.

If you want to debug it, you can easily run it on the CLI via:

unattended-upgrades -d

Logging

If you want to know what was going on, check the logfile located here:

/var/log/unattended-upgrades/

You can also add the following configuration to send you mails when an upgrade occured:

Unattended-Upgrade::Mail "root";
Unattended-Upgrade::MailOnlyOnError "false";

How we do it

Our configuration

For our infrastructure, we’re installing and configuring unattended-upgrades automatically via Ansible on every server. Here’s our default configuration with comments.

20auto-upgrades:

//
// !!! WARNING: This file is managed by Ansible, DON'T EDIT it manually !!!
//

// Enable unattended upgrades.
APT::Periodic::Enable "1";

// Do "apt-get upgrade" every n-days (0=disable).
APT::Periodic::Unattended-Upgrade "1";

// Do "apt-get upgrade --download-only" every n-days (0=disable).
APT::Periodic::Update-Package-Lists "1";

// Do "apt-get upgrade --download-only" every n-days (0=disable).
APT::Periodic::Download-Upgradeable-Packages "1";

// Do "apt-get autoclean" every n-days (0=disable).
APT::Periodic::AutocleanInterval "7";

50unattended-upgrades:

//
// !!! WARNING: This file is managed by Ansible, DON'T EDIT it manually !!!
//

Unattended-Upgrade::Origins-Pattern {
    "o=Debian,n=${distro_codename},l=Debian-Security";
};

Unattended-Upgrade::Package-Blacklist {
};

// Automatically run "dpkg --force-confold --configure -a".
Unattended-Upgrade::AutoFixInterruptedDpkg "true";

// Do automatic removal of new unused dependencies after the upgrade.
Unattended-Upgrade::Remove-Unused-Dependencies "true";

// Install upgrades when the machine is shuting down instead of doing it in the background.
Unattended-Upgrade::InstallOnShutdown "false";

// Automatically reboot if the file /var/run/reboot-required is found after the upgrade.
Unattended-Upgrade::Automatic-Reboot "false";

// Automatically reboot even if there are users currently logged in.
Unattended-Upgrade::Automatic-Reboot-WithUsers "true";

// If automatic reboot is enabled and needed, reboot at the specific time.
Unattended-Upgrade::Automatic-Reboot-Time "now";

// Enable mail notifications.
Unattended-Upgrade::Mail "<our-own-mail>@confirm.ch";
Unattended-Upgrade::MailOnlyOnError "true";

// Enable logging via syslog.
Unattended-Upgrade::SyslogEnable "true";
Unattended-Upgrade::SyslogFacility "daemon";

Security-related vs. non-security-related updates

As you can see, we automatically install security-related updates every day in the background. This ensures our systems get the most important security patches. For the update of every other package, we use an Ansible playbook which simply does a dist-upgrade with a bit of exception handling, followed by running all configuration playbooks again with a final reboot.

Monitor for reboots

We do not automatically reboot our systems, because we don’t want them to go completely down in the middle of the day. Of course you can schedule it by defining the reboot time (other than now), but our approach is a bit different.

We simply monitor for the /var/run/reboot-required file in our monitoring and generate an alert in Zabbix:

Reboot alerts in Zabbix

This generates an alter to our Mattermost monitoring channel, so that everybody in our team knows about the “issue” and a reboot can be scheduled:

Verdict

Combining unattended security upgrades with automated attended full system upgrades is IMHO a nice way to keep your environment up-to-date and secure, without losing control of critical upgrades where a systems engineer is required.

76 Comments

  • ansibleftw

    Hi, seeing your config file reminds me of Debops.
    So, do you use Debops to admin your infrastructure ?

    • Dominique Barton

      Hi. I “know” Debops but nah, we’ve our own Ansible git project and doing configuring & deploying everything via our own Ansible roles. But nice 😉 seeing Debops people out there! 🙂

  • Xavier

    I think there is typos in some places with 50unattended_upgrades in place of 50unattended-upgrades

    • Dominique Barton

      Thanks Xavier. Didn’t saw the code-block titles back then. Fixed it 🙂

  • Palo

    Hi Dominique, thanks a lot for this tutorial.
    Is it possible to setup Unattended upgrades to only report the new available Security updates
    by mail and NOT to install them automatically?

  • Santi

    Hi, thanks for your post, it’s really helpful! Can you explain a little bit more how do you perform the updates of the rest of the packages (the non security ones)? Thank you in advance!

  • Idris

    Can you share the Ansible Playbooks as well 🙂

  • assignment expert

    Although young offenders face adversity for their deeds, the strength perspective views such delinquents as resourceful people in the society.

  • Demolition Joliet

    Automation tools are absolute and necessary for efficiencies.

  • Yasir Ali

    Just want to comment on few general things, the website style is perfect, the written content is really great

  • YasirFactor

    Hi there, You’ve done an incredible job. I will definitely digg it and personally recommend to my friends. I’m sure they will be benefited from this web site.

  • Subway Surfers

    I think they are doing something on this unattended-upgrades right now. Hope they will be more transparent.

  • 192.168.0.1

    Hello! this complex question. Here it is necessary to think…

  • geometry dash

    To get a good blog I think you tried a lot. And I think this is a very interesting blog, it attracts me by the content and creative design.

  • click

    Thank you for updating us.

  • Car Towing

    Never trust a human? I honestly think not everything is meant to be left unattended. You don’t have to sit there waiting for the complete process to finish but human supervision is, in my opinion, something that cannot be excluded just like that.

  • concrete cutting

    This post puts so much faith in AI. Automation is excellent only up to a certain point. There are simply some tasks that a computer cannot do for a human.

  • cherry u

    I’ve tried it! This is very useful! Thanks for this!

  • alishasharma

    this is a very impressive article really I am glad to found this valuable article thanks for sharing this article.

  • Stem Cures

    This has been amazing! 🙂

  • Brian

    Thanks for the post! I definitely am for auto updates. I’m more of a set it and forget it type of guy.

  • Our Site

    Sometimes, using bots will enhance the efficiency of work. But none can defeat the effectiveness and efficiency of human works. If you are looking for any tax services. Just check our website above.

  • mturk.work

    This assumes there aren’t human errors on the update side; automating unattended updates is putting a lot of trust upstream, which isn’t always warranted.

  • Mike

    These are great points. There are a lot of pros and cons with automated updates. Thanks for writing this up!

  • Jim

    This is helpful. Thanks for the code snippets too!

  • remodeling

    Awesome content you shared.

  • Shreya Sharma

    It is one of the most stunning blog kindly see my website and know the more function

  • Neha Rasile

    Once you search of trustworthy escort service supplier there are lots service suppliers who claims to produce best cocotte services, however selecting the proper one may well be extremely difficult. you must be really fastidious and fastidious after you are aiming to avail escort service. little of analysis will certainly assist you to contact the right service provider. Most of the time folks choose to book escort services on-line thus when you are contacting agency or choosing Independent urban center Escorts you wish to travel through the reviews which can positively assist you tochoose the most effective service provider. you must conjointly check their profile on-line in order that you’ll be able to know whom you’re meeting also it’ll help you to grasp a lot of regarding their personality. Before you avail any service you need to check the prices. There are several services that are extremely expensive so it’s necessary to know full details reasonably service you’d be getting. Once you recognize what quantity you have got to pay you can arrange your budget accordingly.

  • Team Flex

    Thanks for sharing this! Helpful to everyone who uses it!

  • Seo

    Thanks for sharing this important information. We’ll have to keep these in mind!

  • lime

    This is amazing

  • Normal

    I love this so much

  • 2nd hand engine

    Much obliged for sharing this significant data. We’ll need to remember these!

  • carl

    Thanks for the great article

  • manesar

    this is a very good post for servuces

  • radio rentals us

    Since 2001, Connect Radio Rental has provided two way radio rentals, Nextel, Verizon & Sprint Push to Talk rentals, Hot Spot Rentals to the Motion Picture, Production, Special Event, Convention, Meeting and Tradeshow industry.

  • Fasgro

    Fasgro is an online specialist grocery platform that delivers fresh products and fresh brands to your door as quickly as 30 minutes within 2 miles of EC2A 4BD.

  • Flash Conveyancing

    I found this page really helpful! Check out our site too!

  • Austin Door Company

    How much time did you put into this? This surely took more than a day to complete, right?

  • 192.168.1.1

    Damm even I am having a similar kind of issue, I have searched all over the internet and even have posted on number of threads on different forum, no solution seems to work.

  • Richmond Tree Trimming

    Always gotta be careful with those auto updates. They can have unintended consequences! Thanks for the good write up!

  • Cuisine Drummond

    En cas de besoin urgent pour une cuisine veuillez confirmer à nos ébénistes un rendez-vous et vous aurez une soumission gratuite!

  • exterior tiling

    The information you bring is very real, reflecting correctly and objectively, it is very useful for grow together. Thank you for sharing…

  • businesses Longmont

    Thanks for contributing the best tips in an approachable, down to earth way.

  • Local Citations

    So much happening here hehe. Hope I could follow this guide step by step. Thanks, bud!

  • Business Listings

    If you will not going to update your website and leaving it without notifying the members, readers, or audiences – that is like abandoning your passion.

  • Ébéniste Laval

    Comme ce blog nous accroissont nos secteurs et nous servons également la région de Laval maintenant.

  • Sarakaur

    Thanks for sharing such a Great Information with us

  • Rénovation Gatineau

    Merci pour ce blog ça m’a donné l’idée d’en créer aussi pour notre nouvelle page!