Operation Details

This section describes the operations that Backrest can be configured to perform on your behalf in detail.

Overview

Backrest executes commands by forking the restic binary. Each Backrest version is validated against a specific version of restic. On startup Backrest searches for a versioned restic in it's data directory (typically ~/.local/share/backrest), followed by /bin/. The restic binary must be named restic-{VERSION}. You can overriderestic command by setting BACKREST_RESTIC_COMMAND env variable when starting Backrest. Otherwise, if no binary is found Backrest will download and install a recent version of restic from restic's github releases. When downloading a restic binary, the download is verified by checking the sha256sum of the downloaded binary against the sha256sum provided by restic and signed by the restic maintainers GPG key.

When running restic commands, Backrest injects the environment variables configured in the repo into the environment of the restic process and it appends the flags configured in the repo to the command line arguments of the restic process. Logs are collected for each command. In the case of an error, Backrest captures the last ~500 bytes of output and displays this directly in the error message (the first and last 250 bytes are shown if the output is longer than 500 bytes). Logs of the command are typically also available by clicking [View Logs] next to an operation, these logs are truncated to 32KB (with the first and last 16KB shown if the log is longer than 32KB).

Types of Operations

Backup

Restic docs on backup

Backups are scheduled under plan settings in the UI. A backup operation creates a snapshot of your data and sends it to a repository. The snapshot is created using the restic backup command.

As the backup runs Backrest will display the progress of the backup operation in the UI. The progress information is parsed from the JSON output of the restic backup command.

The backup flow is as follows

• Hook trigger: CONDITION_SNAPSHOT_START, if any hooks are configured for this event they will run.
  • If any hook exits with a non-zero status, the hook's failure policy will be applied (e.g. cancelling or failing the backup operation).
• The restic backup command is run. The newly craeted snapshot is tagged with the ID of the plan creating it e.g. plan:{PLAN_ID}.
• On backup completion
  • The summary event is parsed from the backup and is stored in the operation's metadata. This includes: files added, files changed, files unmodified, total files processed, bytes added, bytes processed, and most importantly the snapshot ID.
  • If an error occurred: hook trigger CONDITION_SNAPSHOT_ERROR, if any hooks are configured for this event they will run.
  • If successful: hook trigger CONDITION_SNAPSHOT_SUCCESS, if any hooks are configured for this event they will run.
  • Finally CONDITION_SNAPSHOT_END is triggered irrespective of success or failure, if any hooks are configured for this event they will run. This condition is always triggered even if the backup failed.
• If a retention policy is set (e.g. not None) a forget operation is triggered for the backup plan.

Created backups are tagged with some Backrest specific metadata:

• plan:{PLAN_ID} - the ID of the plan that created the snapshot. This is used to group snapshots by plan in the UI.
• created-by:{INSTANCE_ID} - the unique ID of the Backrest instance that created the snapshot. This is used to group snapshots by instance in the UI. Notably, this not necessarily the same as the hostname tracked by restic in the snapshot.

Forget

Restic docs on forget

Forget operations are scheduled by the "forget policy" configured in plan settings in the UI. Forget operations run after backups. A forget operation marks old snapshots for deletion but does not remove data from storage until a prune runs. The forget operation is run using the restic forget --tag plan:{PLAN_ID} command.

Retention policies are mapped to forget arguments:

• By Count maps to --keep-last {COUNT}
• By Time Period maps to the --keep-{hourly,daily,weekly,monthly,yearly} {COUNT} flags

Note that forget will never run for a plan if the forget policy is set to None.

Prune

Restic docs on prune

Prune operations are scheduled under repo settings in the UI. A prune operation removes data from storage that is no longer referenced by any snapshot. The prune operation is run using the restic prune command. Prune operations apply to the entire repo and will show up under the _system_ plan in the Backrest UI.

Prunes are run in compliance with a prune policy which specifies:

• Schedule - the schedule on which to run prune operations. Available types are:
  • Disabled - prune operations are disabled. This means that the repository will grow indefinitely and will never be pruned. You should periodically run a prune operation manually if you choose this option.
  • Cron - a cron expression specifying when to run prune operations.
  • Max Frequency Days - the minimum number of days that must pass between prune operations.
  • Max Frequency Hours - (Advanced) the minimum number of hours that must pass between prune operations. This is useful for running prune operations more frequently than once per day, typically this is a bad idea as prune operations are expensive and should be run infrequently.
• Max Unused Percent - the maximum percentage of the repository that may be left unused after a prune operation runs. Prune operations will try to repack blobs in the repository if more than this percentage of their is unused (e.g. formerly held data belonging to forgotten snapshots).

Check

Restic docs on check

Check operations are scheduled under repo settings in the UI. A check operation verifies the integrity of the repository. The check operation is run using the restic check command. Check operations apply to the entire repo and will show up under the _system_ plan in the Backrest UI.

Checks are run in compliance with a check policy which specifies:

• Schedule - the schedule on which to run check operations. Available types are:
  • Disabled - check operations are disabled. You should periodically run a check operation manually if you choose this option.
  • Cron - a cron expression specifying when to run check operations.
  • Max Frequency Days - the minimum number of days that must pass between check operations.
  • Max Frequency Hours - (Advanced) the minimum number of hours that must pass between check operations. This is useful for running check operations more frequently than once per day, typically this is a bad idea as check operations are expensive and should be run infrequently.
• Read Data % - the percentage of the repository that should be read during a check operation. A value of 100% will check the entire repository which can be very slow and can be expensive if your provider bills for egress bandwidth. Reading back data is intended to verify the integrity of pack files on-disk for potentially unreliable storage (e.g. an HDD running without parity). It typically does not provide much value for a reliable storage provider and can be set to a low percentage or disabled.