pg_probackup备份工具
1.1 pg_probackup备份简介
pg_probackup是管理PG数据库集群备份和恢复的工具。用于执行周期性备份PG实例。兼容PG9.5，,96,10,11,12,13。和其他备份解决方案相比，提供下面优势，帮助部署不同备份策略和处理大量数据。
1.2 pg_probackup功能与限制
1、增量备份。页级别的增量备份，节省磁盘空间，加速备份和转储。有3种不同增量模式，根据数据流部署不同备份策略。
2、增量转储。页级别的转储，在目标目录中重用有效未更改的页面来加速转储。
3、合并。帮助部署”incrementally updated backups”策略，无须进行完备备份。
4、验证。自动数据一致性校验和按需备份校验，无须实际数据恢复。
5、校验。使用checkdb命令按需验证PG实例。
6、保留。根据保留策略管理存档和备份。可以根据恢复实际或要保留的备份数配置保留策略。还可以为特定备份制定生存时间，过期的备份可以合并或者删除。
7、并行化。在多个并行线程上进行备份、恢复、合并、删除、验证和检验。
8、压缩。以压缩状态存储备份数据以节省磁盘空间。
9、消除重复数据。通过不复制未更改的非数据文件如_vm或者_fsm来节省磁盘空间。
10、远程操作。备份位于远程系统上的PG实例或远程恢复备份。
11、从备机上进行备份。通过从备机上进行备份来避免主机上的额外负载。
12、外部目录。备份位于数据目录PGDATA之外的文件和目录，如脚本、配置文件、日志或SQL转储文件。
13、备份Catalog。以纯文本或JSON格式获取备份列表和相应的元信息。
14、归档catalog。以纯文本或JSON格式获取所有WAL时间线和相应元信息的列表。
15、部分还原。仅还原制定的数据库或从转储中排出指定数据库。
管理备份数据：pg_probackup创建一个备份目录。这个目录存储所有备份文件包括额外的元信息，以及WAL归档。可以将不同实例的备份存储在单个目录的不同子目录中。通过该工具可以进行完整备份和增量备份。
1、全量备份，包含从头恢复数据库集群需要的所有数据文件。
2、增量备份。只存储自上次以来更改的数据。允许减小备份大小并加快备份操作。支持以下增量备份模式：
1）PAGE备份。扫描自上次备份以来归档的所有WAL文件。新创建的备份仅包含WAL记录到的页面。如果这些文件大小与数据库集群文件总大小相当，则加速比较小，但备份占用空间仍然较小。
2）DELTA备份。读取PGDATA目录下所有数据文件，仅拷贝自上次备份以来的脏页。无须持续归档。此外，该模式还可以施加与全量备份相同的只读IO压力。
3）PTRACK备份。会动态追踪页面改动。无须持续归档。每次更新数据页时，都会在PTRACK位图标记。由于一个页面只需要PTRACK位图中的一个bit，这个位图非常小。跟踪意味着会有开销，但会显著加速增量备份。
无论选择哪种备份类型，都支持以下WAL传递策略：
1）自主备份，通过复制协议传输将集群恢复到备份时的一致性状态需要的所有WAL文件。即使没有设置连续归档，需要的WAL段也会包含在备份中。
2）归档备份，依赖于连续归档。
限制：备份和转储的PG必须block_size和wal_block_size兼容且有相同主版本号；Windows下不支持ssh远程备份；ssh远程备份时，pg_probackup的版本必须相同。
1.3 pg_probackup安装与使用
wget https://github.com/postgrespro/pg_probackup/archive/2.4.15.tar.gz
tar -zxvf 2.4.15.tar.gz
cd pg_probackup-2.2.7/
make USE_PGXS=1 PG_CONFIG=/data/postgresql/bin/pg_config top_srcdir=/home/postgres/postgresql-13.3
make USE_PGXS=1 PG_CONFIG=/data/postgresql/bin/pg_config top_srcdir=/home/postgres/postgresql-13.3 install一旦安装了pg_probackup，请完成以下设置。
1.4 pg_probackup初始化备份目录
所谓初始化：就是创建一个备份目录；用于存放备份文件和归档日志文件。
初始化备份目录，请运行以下命令：
pg_probackup init -B backup_dir
-- 示例
pg_probackup init -B /data/postgres/probackuppg_probackup创建备份目录，其中包含以下子目录：
— wal/-wal文件目录。
— backups/-备份文件的目录。
备份目录初始化了就可以添加备份实例。
1.5 向备份目录添加新的备份Instance
pg_probackup可以在一个备份目录中存储多个数据库集群的备份。要设置所需的子目录，必须将备份实例添加到要备份的每个数据库群集的备份目录中。
要添加新的备份实例，请运行以下命令：
pg_probackup add-instance -B backup_dir -D data_dir --instance instance_name [remote_options]
-- 本地添加备份instance 示例
pg_probackup add-instance -B /data/postgres/probackup -D /data/postgres/data --instance local_6000— data_dir是要备份的群集的数据目录。要设置和使用pg_probackup，需要对此目录的写访问权限。
— instance_name是将存储此群集的WAL和备份文件的子目录的名称。
— remote_options是可选参数，仅当data_dir位于远程系统上时，才需要指定这些参数。后续在详细讲解，参考。
pg_probackup在备份目录的 backups 和 wal 目录下创建 instance_name 子目录。backups/instance_name 目录包含 pg_probackup.conf 控制此备份实例的pg_probackup设置的配置文件。如果使用remote_选项运行此命令，则指定的参数将添加到pg_probackup.conf。
1.6 配置PostgreSQL集群
如果使用非postgres用户需要配置以下权限
For PostgreSQL 10 or higher:
$ psql -p 5432 -U postgres
BEGIN;
CREATE ROLE backup WITH LOGIN REPLICATION;
GRANT USAGE ON SCHEMA pg_catalog TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.current_setting(text) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_is_in_recovery() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_start_backup(text, boolean, boolean) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_stop_backup(boolean, boolean) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_create_restore_point(text) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_switch_wal() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_last_wal_replay_lsn() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_current() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_current_snapshot() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_snapshot_xmax(txid_snapshot) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_control_checkpoint() TO backup;
COMMIT;1.7 配置postgresql.conf
— 确保wal_level参数高于minimal。
— 如果要在primary上配置存档，则存档模式必须设置为“on”或“always”。要在standby下执行存档，请将此参数设置为“always’
— 设置archive_命令参数，如下所示：
archive_command = 'install_dir/pg_probackup archive-push -B backup_dir --instance instance_name --wal-file-name=%f [remote_options]'
-- 示例
/opt/pg13/bin/pg_probackup archive-push -B /data/postgres/probackup --instance local_6000 --wal-file-path=%p
-- wal-file-name=%f1.8 创建全量备份
创建备份，命令
pg_probackup backup -B backup_dir --instance instance_name -b backup_mode示例
pg_probackup backup -B /data/postgres/probackup --instance local_6000 -b full1.9 查看备份
pg_probackup show -B /data/postgres/probackup
pg_probackup show -B /data/postgres/probackup --instance local_6000 -i QGQMKH
pg_probackup show -B /data/postgres/probackup --instance local_6000 --archive1.10 基于PITR备份与恢复
-- 备份
pg_probackup backup -B /data/postgres/probackup --instance local_6000 -b full
--恢复之前需要先停库；在清理PGDATA目录以及外在的表空间目录
pg_ctl stop;
rm -rf $PGDATA/*
--恢复到新的PGDATA目录
pg_probackup restore -B /data/postgres/probackup --instance local_6000 -D /data/postgres/data6000 --recovery-target-time='2021-07-11 11:05:17'
11:05:17'
-- 恢复到原目录
pg_probackup restore -B /data/postgres/probackup --instance local_6000 --recovery-target-time='2021-07-11 11:05:17'1.11 增量备份
-- 增量备份DELTA
pg_probackup backup -B /data/postgres/probackup --instance local_6000 -b delta
-- 增量备份page
pg_probackup backup -B /data/postgres/probackup --instance local_6000 -b page
-- Postgres Pro Standard : Documentation: 13: pg_probackup1.12 远程备份
--set up ssh
ssh-copy-id postgres@node2
--备份库 添加备份实例
pg_probackup add-instance -B /data/postgres/probackup -D /data/postgres/data --instance pg207_6000 --remote-proto=ssh --remote-host=192.168.127.126 --remote-port=22 --remote-user=postgres --remote-path=/opt/pg13/bin --ssh-options='-o ServerAliveCountMax=5 -o ServerAliveInterval=60'
--案例库 配置的参数
--备份需要用到postgres超级用户；所以在pg_hba.conf;
host all postgres 192.168.127.129/24 trust
-- postgresql.conf
wal_level = replica
archive_mode = on
archive_command = '/opt/pg13/bin/pg_probackup archive-push -B /data/postgres/probackup --instance pg207_6000 --wal-file-path=%p --wal-file-name=%f --remote-proto=ssh --remote-host=192.168.127.129 --remote-port=22 --remote-user=postgres --remote-path=/opt/pg12/bin'
-- 备份库； 执行远程备份
pg_probackup backup -B /data/postgres/probackup --instance pg207_6000 -b full --remote-proto=ssh --remote-host=192.168.127.126 --remote-port=22 --remote-user=postgres --remote-path=/opt/pg12/bin --ssh-options='-o ServerAliveCountMax=5 -o ServerAliveInterval=60'
--Take a FULL backup:
pg_probackup-11 backup -B /mnt/backups --instance 'pg-11' -b FULL --stream --remote-host=postgres_host --remote-user=postgres -U backup -d backupdb
--Take an incremental backup in the DELTA mode:
pg_probackup-11 backup -B /mnt/backups --instance 'pg-11' -b delta --stream --remote-host=postgres_host --remote-user=postgres -U backup -d backupdb
--
pg_probackup-11 set-config -B /mnt/backups --instance 'pg-11' --remote-host=postgres_host --remote-user=postgres -U backup -d backupdb
pg_probackup-11 backup -B /mnt/backups --instance 'pg-11' -b delta --stream
--pg_probackup-11 show-config -B /mnt/backups --instance 'pg-11'
--pg_probackup-11 show -B /mnt/backups --instance 'pg-11'
-- 备份库；查看备份
pg_probackup show -B /data/postgres/probackup --instance pg207_60001.13 远程恢复
-- 恢复
pg_probackup restore -B /data/postgres/probackup --instance local_6000 --recovery-target-time='2021-07-11 14:54:55'1.14 delta模式和page模式的备份性能差异
测试目的：
本次测试pg_probackup工具的delta模式和page模式的备份性能差异
测试准备:
准备10亿条数据(数据大小146G)：
pgbench -i -s 10000 -U backup -h 127.0.0.1 pgbench
pgbench=# \dt
List of relations
Schema | Name | Type | Owner
--------+------------------+-------+--------
public | pgbench_accounts | table | backup
public | pgbench_branches | table | backup
public | pgbench_history | table | backup
public | pgbench_tellers | table | backup
(4 rows)
pgbench=# select count(*) from pgbench_accounts;
count
------------
1000000000
(1 row)
磁盘占用：
[postgres@ecos75]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.5G 0 7.5G 0% /dev
tmpfs 7.5G 12K 7.5G 1% /dev/shm
tmpfs 7.5G 9.2M 7.5G 1% /run
tmpfs 7.5G 0 7.5G 0% /sys/fs/cgroup
/dev/mapper/vg00-lv_root 20G 9.9G 8.8G 53% /
/dev/sda1 501M 335M 130M 73% /boot
/dev/sda2 524M 12K 524M 1% /boot/efi
/dev/sdb1 296G 147G 134G 53% /postgresql/data
/dev/sdc1 296G 65M 281G 1% /postgresql/backup
tmpfs 1.5G 0 1.5G 0% /run/user/0
/dev/sdd1 296G 122G 160G 44% /postgresql/archive全量备份（DELTA）
pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup --stream -b full
[postgres@ecos75]$ pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup --stream -b full --temp-slot
INFO: Backup start, pg_probackup version: 2.4.15, instance: pgbak, backup ID: QLOIL1, backup mode: FULL, wal mode: STREAM, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
INFO: PGDATA size: 146GB
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 33m:19s
2021-07-11 16:12:20.388 CST [24069] LOG: restore point 'pg_probackup, backup_id QLOIL1' created at 23/C0000178
2021-07-11 16:12:20.388 CST [24069] STATEMENT: SELECT pg_catalog.pg_create_restore_point($1)
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 15s
INFO: Validating backup QLOIL1
INFO: Backup QLOIL1 data files are valid
INFO: Backup QLOIL1 resident size: 146GB
INFO: Backup QLOIL1 completed
时间:2021-07-11 15:39 ~ 2021-07-11 16:31ps:出现如下错误，需要配置pg lib库环境变量：
export LD_LIBRARY_PATH=/usr/local/postgresql-13.3/lib
ERROR: could not connect to database postgres: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket '/var/run/postgresql/.s.PGSQL.5432'?增量数据准备：
插入一亿条增量数据（约15G数据）
pgbench -i -s 1000 -U probackup -h 127.0.0.1 pgbench_inc
磁盘占用：
[postgres@ecos75]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.5G 0 7.5G 0% /dev
tmpfs 7.5G 12K 7.5G 1% /dev/shm
tmpfs 7.5G 9.1M 7.5G 1% /run
tmpfs 7.5G 0 7.5G 0% /sys/fs/cgroup
/dev/mapper/vg00-lv_root 20G 9.9G 8.8G 53% /
/dev/sda1 501M 335M 130M 73% /boot
/dev/sda2 524M 12K 524M 1% /boot/efi
tmpfs 1.5G 0 1.5G 0% /run/user/0
/dev/sdb1 296G 162G 119G 58% /postgresql/data
/dev/sdc1 296G 147G 134G 53% /postgresql/backup
/dev/sdd1 296G 13G 268G 5% /postgresql/archive增量备份（DELTA）
pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup --stream -b delta --temp-slot
[postgres@ecos75]$ pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup --stream -b delta --temp-slot
INFO: Backup start, pg_probackup version: 2.4.15, instance: pgbak, backup ID: QLOMCV, backup mode: DELTA, wal mode: STREAM, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
INFO: Parent backup: QLOIL1
INFO: PGDATA size: 161GB
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 31m:29s
2021-07-11 17:32:00.310 CST [30184] LOG: restore point 'pg_probackup, backup_id QLOMCV' created at 26/C1000178
2021-07-11 17:32:00.310 CST [30184] STATEMENT: SELECT pg_catalog.pg_create_restore_point($1)
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 0
INFO: Validating backup QLOMCV
INFO: Backup QLOMCV data files are valid
INFO: Backup QLOMCV resident size: 15GB
INFO: Backup QLOMCV completed
时间：2021-07-11 17:00 ~ 2021-07-11 17:35全量备份（PAGE）
修改postgresql.conf
vi postgresql.conf
archive_command='pg_probackup archive-push -B /postgresql/backup --instance pgbak --wal-file-name=%f'
删除增量库pgbench_inc
postgres=# drop database pgbench_inc;
pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup -b full
[postgres@ecos75]$ pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup -b full
INFO: Backup start, pg_probackup version: 2.4.15, instance: pgbak, backup ID: QLILZE, backup mode: FULL, wal mode: ARCHIVE, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
INFO: Wait for WAL segment /postgresql/backup/wal/pgbak/0000000100000020000000AE to be archived
INFO: PGDATA size: 146GB
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 38m:52s
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executedv
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 14s
INFO: Validating backup QLILZE
INFO: Backup QLILZE data files are valid
INFO: Backup QLILZE resident size: 146GB
INFO: Backup QLILZE completed
时间：2021-07-12 08:57 ~ 2021-07-12 09:54增量数据准备:
插入一亿条增量数据（约15G数据）
pgbench -i -s 1000 -U probackup -h 127.0.0.1 pgbench_inc磁盘占用：
[postgres@ecos75]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.5G 0 7.5G 0% /dev
tmpfs 7.5G 12K 7.5G 1% /dev/shm
tmpfs 7.5G 18M 7.5G 1% /run
tmpfs 7.5G 0 7.5G 0% /sys/fs/cgroup
/dev/mapper/vg00-lv_root 20G 9.9G 8.8G 53% /
/dev/sda1 501M 335M 130M 73% /boot
/dev/sda2 524M 12K 524M 1% /boot/efi
/dev/sdb1 296G 162G 119G 58% /postgresql/data
/dev/sdc1 296G 159G 122G 57% /postgresql/backup
tmpfs 1.5G 0 1.5G 0% /run/user/0
/dev/sdd1 296G 122G 159G 44% /postgresql/archive增量备份（PAGE）
pg_probackup-12 backup -B /postgresql/backup/ --instance pgbak -b PAGE -U probackup
[postgres@ecos75]$ pg_probackup backup -B /postgresql/backup/ --instance pgbak -b page -j 4 -U probackup
INFO: Backup start, pg_probackup version: 2.4.15, instance: pgbak, backup ID: QLPZ42, backup mode: PAGE, wal mode: ARCHIVE, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
INFO: Wait for WAL segment /postgresql/backup/wal/pgbak/0000000100000029000000C8 to be archived
INFO: Parent backup: QLPUN6
INFO: PGDATA size: 161GB
INFO: Extracting pagemap of changed blocks
INFO: Pagemap successfully extracted, time elapsed: 142 sec
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 5m:16s
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 8s
INFO: Validating backup QLPZ42
INFO: Backup QLPZ42 data files are valid
INFO: Backup QLPZ42 resident size: 15GB
INFO: Backup QLPZ42 completed
时间：2021-07-12 10:33 ~ 2021-07-12 10:44测试结果：
本次只提供了测试流程和其中一次测试结果：
模式DELTAPAGE全量备份数据量：146G/10亿 总耗时：52min 数据传输：34min 数据验证：18min数据量：146G/10亿 总耗时：57min 数据传输：39min 数据验证：18min增量备份数据量：15G/1亿 总耗时：35min 数据传输：31min 数据验证：4min数据量：15G/1亿 总耗时：11min 数据传输：8min 数据验证：3min
通过本次测试发现，delta模式与page模式全量备份差异不明显，增量备份方面，page模式比delta模式备份效率高3倍（在这个数据量级下，数据量越大，差异或许更大）
原创文章，作者：奋斗，如若转载，请注明出处：https://blog.ytso.com/237280.html