BTRFS Rotating Snapshots

Python Script for Roating Snapshots

By Andrew Que

About

The file system BTRFS has the ability do to snapshots. This can be useful for both backups and as a way to preserve history. BTRFS Rotating Snapshots is a Python script designed to simplify maintaining rotating snapshots.

BTRFS does the majority of the work for snapshot rotation and this script just makes it easier.

How it works

BTRFS Rotating Snapshots has three primary functions:

Typically snapshots are setup to be taken at regular intervals (hourly, daily monthly) using a cron job.

Taking Snapshots

There are two ways to specify commands: with inline parameters, and using a configuration file.

root@system:/# btrfsSnapshots.py take /path/to/subvolume /path/to/snapshots root@system:/# btrfsSnapshots.py --config /path/to/config.json take

This function will take a snapshot of the subvolume /path/to/subvolume and place the snapshot subvolume in /path/to/snapshots.

Timestamp format

Snapshots are named with the time the snapshot was taken. The date format is variable but typically is in the format YYYY-MM-DD HH:MM:SS.

The timestamp has three configuration variables:

date_format and snapshot_name_format often match one another. However, the snapshot name format might force some fields. For example, %Y-%m-%d %H:%M:00 forces the second field to zero. This can be desirable for cron jobs as they sometimes take a second or two to launch the scheduled process.

Another common timestamp is to omit the time of day completely. For example, %Y-%m-%d could be used if snapshots are taken at most once a day.

The regular expression is used to get a directory listing of existing snapshots. Other than the snapshot, BTRFS Rotation Snapshots is stateless. So to find the existing snapshots (needed during pruning) the existing listing of the destination directory is used and matched against a regular expression.

One side effect is that snapshots not following the normal pattern are not removed. This can be useful for named snapshots. However, this too can be overridden.

Checking for changes

By default new snapshots are checked against the previous snapshot for changes. If no changes have occurred, the new snapshot is removed. Checking for changes uses rsync.

Pruning Snapshots

Two forms of pruning: all configured by the command line, and from a configuration file.

root@system:/# btrfsSnapshots.py --keep-last-days=10 --keep-last-months=-1 prune /path/to/snapshots root@system:/# btrfsSnapshots.py --config /path/to/config.json prune

Periodically snapshots should be pruned so that unneeded snapshots are removed. There are four variables that control how long snapshots remain:

Configuration is saved in a JSON file. Anything that can be specified from the command line can be specified in the configuration file. Only difference is that underscores change to dashes.

{
  "keep_last_snapshots" : 11,
  "keep_last_days"      : 7,
  "keep_last_months"    : 5,
  "keep_last_years"     : 3
}

This configuration will yield a result like this:

root@system:/snapshots# ls -l
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2019-12-31 17:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2020-12-31 17:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2021-12-31 17:45:00'
 drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-05-31 17:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-06-30 17:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-07-31 17:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-08-31 17:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-09-30 17:45:00'
 drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-20 17:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-21 17:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-22 17:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-23 17:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-24 17:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-25 17:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-26 17:45:00'
 drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-27 15:15:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-27 15:30:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-27 15:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-27 16:00:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-27 16:15:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-27 16:30:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-27 16:45:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-27 17:00:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-27 17:15:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-27 17:30:00'
drwxr-xr-x 1 root root 54 Jan  1 00:00 '2022-10-27 17:45:00'
 -rw-r--r-- 1 root root 32 Oct 27 17:45 lastSnapshot.txt

In this example snapshots are taken every 15 minutes. This is what remains after 5 years. The backgrounds are color coated:

A value of 0 in one of the fields means no snapshots in that interval are kept. A value of -1 means keep all records. For example, -1 in months will keep one snapshot for every month. All settings after this are moot. So if months are set to -1, it doesn't matter what is put in year because all months are saved anyway.

Cron job

Generally this script will be run using a cron job. There are two different jobs, taking a snapshot and pruning the snapshots.

# Hourly snapshots
0 * * * *     /usr/local/sbin/btrfsSnapshots.py --config=/etc/btrfsSnapshot/subvolumeA.json take
0 * * * *     /usr/local/sbin/btrfsSnapshots.py --config=/etc/btrfsSnapshot/subvolumeB.json take

# End-of-day snapshots
55 23 * * *   /usr/local/sbin/btrfsSnapshots.py --config=/etc/btrfsSnapshot/subvolumeA.json take -f

# Prune snapshots
0 0 * * *     /usr/local/sbin/btrfsSnapshots.py --config=/etc/btrfsSnapshot/subvolumeA.json prune
0 0 * * *     /usr/local/sbin/btrfsSnapshots.py --config=/etc/btrfsSnapshot/subvolumeB.json prune

In the example above, hourly snapshots are taken on two subvolumes.

BTRFS Setup

While snapshots can be used to make images of an entire hard drive, subvolumes can be used to effectively make backup images of individual directories.

For this example, say we want to setup snapshots on the following sets of data:

Assume the root file system is BTRFS and none of these directories currently exist. If they do, rename them so they can be created as subvolumes.

Create the subvolumes.

root@system:/# btrfs subvolume create /home/user/Documents
Create subvolume '/home/user/Documents'
root@system:/# btrfs subvolume create /home/user/Photos
Create subvolume '/home/user/Photos'
root@system:/# btrfs subvolume create /home/user/Recordings
Create subvolume '/home/user/Recordings'
root@system:/# btrfs subvolume create /home/user/Source
Create subvolume '/home/user/Source'

Now move your data into these directories. Once populate with data, you can use BTRFS Rotating Snapshots on those subvolumes. You will just need a path to place the snapshots. Some systems use a hidden snapshot directory like ~/.snapshots. If not wanting to hide the snapshots you can use ~/snapshots. To get the snapshots to show up as the first item in a directory listing, use ~/@snapshots. To make it the last item, use ~/_snapshots.

Download

Archives signed by Andrew Que. Keys are generated yearly and can be found on the MIT PGP Public Key Server by doing a search for Andrew Que. They can also be downloaded directly from DrQue.net.

Version 1.0.1

Released November 06, 2022.

Bug fix to sending snapshots to remote drive.

Source code

Source SHA256: dc52dd8047dd3a7ea2b0dfa05d3fbc5c1568a83d7d198b5c05d58101ef031d8f

Source PGP signature

Version 1.0.0

Released November 01, 2022.

Source code

Source SHA256: 040723001a63b233cd5f7e19177bdf8ae505cff6cf4525aec56ed4c515f82383

Source PGP signature

User comments

Feel free to leave some feedback. A subset of Markdown is supported.

Opinions are not censored, but all advertisements will unceremoniously be deleted.

Comments are Javascript-based. Without Javascript enabled, this message is all that appears.

Copyright

BTRFS Rotating Snapshots is free, open-source software released under the GNU General Public License v3.

Author

BTRFS Rotating Snapshots is written and maintained by Andrew Que. To get in touch with Andrew Que, visit his contact page.