Working around the read-only file systems in CoreOS with overlay

I had a specific use case to place the quiesce scripts on the CoreOS running in a VMware virtual machine, so that I could take a consistent backup with Veaam.

While I generally agree this is a bad idea, and I admit that I store most of the important stuff in git, there are times when I am lazy in development and just want to have a backup of any sort.

So right back to the subject, shall we?

Of course building own image and keeping it up to date is one of the options, but let’s call it a plan Z for the moment.

Luckily, an overlay mounts can be used to work around the fact that /usr is a read-only partition.

I decided to keep the scripts ion /opt/sbin (as this location is read-write and persists reboots).

It is as simple as:

mkdir /opt/sbin
mount -o "lower=/usr/sbin:/opt/sbin" -t overlay overlay /usr/sbin

Also in order to survive the reboots we need the following systemd unit:

[Unit]
Description=Overlay mount /usr/sbin mount
Before=local-fs.target
ConditionPathExists=/opt/sbin

[Mount]
Type=overlay
What=overlay
Where=/usr/sbin
Options=lowerdir=/usr/sbin:/opt/sbin

[Install]
WantedBy=local-fs.target

Finally here are my quiesce tools that I use.

The /usr/sbin/pre-freeze-script script shuts down all the docker containers.

$ cat /usr/sbin/pre-freeze-script
#!/bin/bash
docker stop $(docker ps -aq) >/dev/null 2>&1

The /usr/sbin/post-thaw-script script restarts docker.service. This forces all containers to start up in the right order (think legacy links). I attempted to write logic to start them containers without service restart, but that became pretty complex code with no added benefit so I just gave up.

$ cat /usr/sbin/post-thaw-script
#!/bin/bash
systemctl restart docker.service >/dev/null 2>&1