Transactional Systems: Examples Core OS / RedHat: Various: SUSE: - PowerPoint PPT Presentation

Transactional Systems: Examples Core OS / RedHat: Various: SUSE:

Common Properties of Transactional Systems Transactional / atomic updates • Read-only root file system • Focus on large deployments (clouds) or embedded systems • Minimal base system • Automatic updates / reboots • Integrity protection

What Is a Transactional Update? A Transactional update is an update that Is atomic ● – Either fully applied or not applied at all – Update does not influence the running system Can be rolled back ● – A failed or incompatible update can be quickly discarded to restore the previous system condition – ➔ This definition is valid for all atomic distributions ➔ Atomicity guaranteed by rebooting the system ➔ Apart from that, implementations vary widely

Transactional Updates in the SUSE World

SUSE Tools ● Zypper for package management ● Snapper for snapshot creation ● Btrfs as default file system

Updates with snapper Active system 1. Create “pre” snapshot current Backup 2. Update the current system / / (pre) 3. Create “post” snapshot Update is modifying the currently • Backup active file system / (post) Restarts services immediately •

Updates with snapper A Transactional Update is an update that ● Is atomic – Either fully applied, or not applied at all – Update does not influence the running system ● Can be rolled back – A failed or incompatible update can be quickly discarded to restore the previous system condition

Updates with transactional-update 1. Create new target snapshot 2. Update system, make snapshot Active system read-only and set as default for next boot ● Current root file system is not new current modified / (ro) / ● No backup snapshots required, due to ro system ● Will not do anything if there are no updates ● Discard snapshot in case of errors

Updates with snapper A Transactional Update is an update that ● Is atomic – Either fully applied, or not applied at all – Update does not influence the running system ● Can be rolled back – A failed or incompatible update can be quickly discarded to restore the previous system condition ➢ Reboot will provide atomicity

Helper Applications: health-checker ● Add your own checker scripts to check for system consistency ● Automatic rollback if checks fail on first boot ● GRUB2: Select old kernel if system wasn’t able to complete initrd current new / (ro) /

Comparison & Usage

Comparison to Other Read-Only Systems ● No separate tooling or infrastructure needed ● Works with any standards-compliant RPM package ● General-purpose tool: especially useful for servers and clusters ● Fast snapshot switching ● Sane /etc and /var handling ● Only works with BTRFS root file systems

Live Demo

Cheat Sheet Transactional Updates List repositories List snapshots zypper lr -d snapper list zypper lr -d snapper list Refresh repositories Mark snapshots for removal by snapper zypper ref zypper ref transactional-update cleanup transactional-update cleanup Update installed packages View default subvolume transactional-update up transactional-update up btrfs subvolume get-default / btrfs subvolume get-default / Perform a distribution update Open shell transactional-update dup transactional-update dup transactional-update shell transactional-update shell Request reboot Install package(s) transactional-update pkg in <name> transactional-update pkg in <name> transactional-update reboot transactional-update reboot Update package(s) System rollback transactional-update pkg up <name> transactional-update pkg up <name> transactional-update rollback [number] transactional-update rollback [number] Remove package(s) transactional-update pkg rm <name> transactional-update pkg rm <name>

A Deeper Look

Handling of Special Directories Writable directories on an otherwise read-only system: ● /var ● /etc Snapshots are only taken of the root file system by default!

/var handling /var handling ● /var is a special directory because it contains variable data ● Has to have read-write permissions ● Cannot be rolled back ● A rollback would usually delete production data (e.g., databases or Docker images) ● Typically stored on a separate subvolume or partition ● /var will not be mounted into the update snapshot (i.e. packages can not modify it) ● Special handling: directories will be created on next boot ● Files can be created with systemd-tmpfiles; for more complex initialization or migration tasks, use systemd services on first boot

/etc handling ● On read-only systems, /etc has to be writable ● Mounted as an overlay file system ● Overlays stored in /var/lib/overlays ● Each snapshot has one corresponding /etc overlay Overlay will be mounted during the update ● Before: Only one overlay for all snapshots ● ● /etc overlays are stacked ● Changes accumulated before the next boot will be visible both in the current and the next snapshot (if not changed in both overlays) ● On rollback, only the layers up to that snapshot will be mounted

/etc handling ove ove r r l l a a y y / / e e t t c c ove ove r r l l a a y y de de f f a a ul ul t t s s , , uppe uppe r r di di r r =/ =/ s s ys ys r r oot oot / / va va r r / / l l i i b/ b/ ove ove r r l l a a y/ y/ 18/ 18/ e e t t c c , , l l owe owe r r di di r r =/ =/ s s ys ys r r oo oo f f or or =/ =/ va va r r / / l l i i b/ b/ ove ove r r l l a a y y , , x- x- s s ys ys t t e e m d. m d. r r e e qui qui r r e e s s - - m ount m ount s s - - f f or or =/ =/ s s ys ys r r oot oot / / va va r r , , x- x- s s ys ys t t e e m d. m d. r r e e qui qui r r e e s s - - m ount m ount s s - - f f or or =/ =/ s s ys ys r r oot oot / / va va r r / / l l i i b/ b/ ove ove r r l l a a y y To prevent the overlayfs stack from growing infinitely, the /etc state of ● the oldest available snapshot is synchronized into the new snapshot The /etc directory in a snapshot (/.snapshots/<num>/snapshot/etc) ● alone does not represent a consistent system state! Do not mess with the files in /var/lib/overlays (although, you can use ● them as a backup source if you want)

Other Directories ● /opt , /var/log and /boot/grub2 will be bind mounted into the update snapshot ● Everything else, including /srv , won’t! ➔ Packages have to follow the FHS and packaging guidelines

Mount Summary Bind mounts: /srv /bo /var/ /etc Active system / (ro) /var /home /opt grub log layer x ... arc /bo /var/ Visible during /etc / /opt grub update log layer x+1 arc

Utilities

Helper Applications: rebootmgr ● transactional-update.timer triggers daily update including reboot ● rebootmgr manages reboot (e.g. in maintenance windows or synchronized via etcd) ● SUSE CaaS Platform users: not used there; use Velum to trigger cluster-wide reboots instead

Other ● Configuration file: /etc/transactional-update.conf ● Snapper will clean up old snapshots ● transactional-update is the only way to update a read-only system ● Zypper integration is planned

Pitfalls ● Snapshots will be branched from the current system → snapshots will not contains the previous snapshot’s contents if the system hasn’t been rebooted! ● When using transactional-update on a read-write system → Don’t forget to reboot your system before making any changes to the root file system!

What’s Next?

Availability SUSE Linux Enterprise Server Service Pack 1: ● Technology Preview ● Compatible with most packages (~30 packages left out of 10,000) ● See documentation for a list of unsupported packages

Availability

Future Development ● Integration into SUSE Linux Enterprise Server ● IMA / EVM support for system verification / integrity ● Integrate transactional-update as zypper plugin ● Fix remaining RPM packages with scripts modifying /var and /srv